Jul 30 08:11:45.626658 ip-10-0-136-119 systemd[1]: Starting Kubernetes Kubelet... Jul 30 08:11:46.020179 ip-10-0-136-119 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 08:11:46.020179 ip-10-0-136-119 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 30 08:11:46.020179 ip-10-0-136-119 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 08:11:46.020179 ip-10-0-136-119 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 30 08:11:46.020885 ip-10-0-136-119 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 08:11:46.022508 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.022389 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 30 08:11:46.031635 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031615 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 08:11:46.031635 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031631 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 08:11:46.031635 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031635 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031640 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031643 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031646 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031649 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031652 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031655 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031658 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031661 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031664 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031666 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031669 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031672 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031674 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031677 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031681 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031685 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031687 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031696 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031699 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 08:11:46.031739 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031701 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031704 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031707 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031709 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031712 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031715 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031717 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031719 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031722 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031724 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031727 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031729 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031732 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031734 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031738 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031740 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031744 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031747 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031749 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 08:11:46.032221 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031752 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031756 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031760 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031763 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031765 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031768 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031771 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031774 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031777 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031779 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031782 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031785 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031787 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031790 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031793 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031795 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031798 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031800 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031803 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 08:11:46.032702 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031805 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031807 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031810 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031813 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031816 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031818 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031821 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031825 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031829 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031832 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031834 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031838 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031840 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031844 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031846 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031849 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031851 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031854 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031856 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031859 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 08:11:46.033205 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031862 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031865 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031869 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031872 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031875 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031877 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031880 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031883 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031885 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031888 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031890 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031893 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031896 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031898 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031900 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031903 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031907 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031909 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.031912 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032613 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 08:11:46.033727 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032621 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032623 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032626 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032629 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032632 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032636 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032638 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032641 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032644 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032647 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032649 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032652 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032654 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032657 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032659 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032662 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032665 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032668 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032670 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 08:11:46.034212 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032673 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032676 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032678 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032681 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032683 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032688 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032691 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032695 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032697 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032700 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032704 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032706 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032709 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032712 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032714 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032717 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032719 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032722 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032725 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032728 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 08:11:46.034703 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032731 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032734 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032736 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032738 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032741 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032743 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032747 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032750 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032752 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032755 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032758 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032760 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032763 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032767 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032771 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032774 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032777 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032780 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032783 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 08:11:46.035191 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032785 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032788 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032791 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032794 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032796 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032799 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032802 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032807 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032810 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032814 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032816 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032819 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032822 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032825 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032827 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032830 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032832 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032834 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032837 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 08:11:46.035679 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032839 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032842 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032845 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032847 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032851 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032853 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032856 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032858 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032861 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032864 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032866 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032869 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032871 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032874 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032876 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032879 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032882 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032884 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032887 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032890 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 08:11:46.036142 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.032892 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033287 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033293 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033296 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033299 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033306 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033309 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033312 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033315 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033317 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033320 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033324 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033327 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033330 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033333 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033336 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033339 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033342 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033344 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 08:11:46.036779 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033347 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033350 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033353 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033356 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033358 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033362 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033364 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033367 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033385 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033388 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033391 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033393 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033396 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033399 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033401 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033404 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033406 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033409 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033411 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 08:11:46.037249 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033415 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033418 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033421 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033431 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033435 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033437 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033440 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033442 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033445 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033448 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033451 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033454 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033456 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033459 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033462 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033466 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033468 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033471 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033473 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033476 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 08:11:46.037753 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033478 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033481 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033483 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033486 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033489 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033491 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033494 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033497 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033499 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033501 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033504 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033506 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033509 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033512 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033515 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033517 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033520 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033523 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033525 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033528 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 08:11:46.038267 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033530 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033533 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033536 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033539 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033542 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033544 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033547 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033549 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033552 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033555 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033558 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033560 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033563 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033565 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033568 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033570 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033573 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033575 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033578 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033580 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 08:11:46.038775 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033584 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.033586 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033660 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033671 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033686 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033693 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033700 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033705 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033712 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033716 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033720 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033723 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033726 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033730 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033734 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033738 2576 flags.go:64] FLAG: --cgroup-root="" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033741 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033744 2576 flags.go:64] FLAG: --client-ca-file="" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033746 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033749 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033753 2576 flags.go:64] FLAG: --cluster-domain="" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033756 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033759 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033762 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 30 08:11:46.039283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033766 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033770 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033773 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033776 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033779 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033782 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033785 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033787 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033791 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033794 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033798 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033801 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033803 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033806 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033809 2576 flags.go:64] FLAG: --enable-server="true" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033812 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033816 2576 flags.go:64] FLAG: --event-burst="100" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033819 2576 flags.go:64] FLAG: --event-qps="50" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033822 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033826 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033829 2576 flags.go:64] FLAG: --eviction-hard="" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033832 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033838 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033841 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033844 2576 flags.go:64] FLAG: --eviction-soft="" Jul 30 08:11:46.039873 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033847 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033850 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033853 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033856 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033859 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033862 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033865 2576 flags.go:64] FLAG: --feature-gates="" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033868 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033871 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033874 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033877 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033880 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033883 2576 flags.go:64] FLAG: --help="false" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033886 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033889 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033892 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033895 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033898 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033902 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033904 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033907 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033909 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033912 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 30 08:11:46.040545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033916 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033919 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033922 2576 flags.go:64] FLAG: --kube-reserved="" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033925 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033928 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033931 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033936 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033939 2576 flags.go:64] FLAG: --lock-file="" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033941 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033944 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033947 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033953 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033955 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033958 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033961 2576 flags.go:64] FLAG: --logging-format="text" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033964 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033967 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033970 2576 flags.go:64] FLAG: --manifest-url="" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033973 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033977 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033980 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033984 2576 flags.go:64] FLAG: --max-pods="110" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033988 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033990 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033993 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 30 08:11:46.041143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033996 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.033999 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034002 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034005 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034012 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034015 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034018 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034021 2576 flags.go:64] FLAG: --pod-cidr="" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034025 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034030 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034033 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034037 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034039 2576 flags.go:64] FLAG: --port="10250" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034043 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034047 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d6222cc66f8c99f9" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034050 2576 flags.go:64] FLAG: --qos-reserved="" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034053 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034056 2576 flags.go:64] FLAG: --register-node="true" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034059 2576 flags.go:64] FLAG: --register-with-taints="" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034063 2576 flags.go:64] FLAG: --registry-burst="10" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034066 2576 flags.go:64] FLAG: --registry-qps="5" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034068 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034071 2576 flags.go:64] FLAG: --reserved-memory="" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034075 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034078 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 30 08:11:46.041825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034081 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034084 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034087 2576 flags.go:64] FLAG: --runonce="false" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034090 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034093 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034096 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034099 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034101 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034104 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034108 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034111 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034114 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034116 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034119 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034122 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034125 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034129 2576 flags.go:64] FLAG: --system-cgroups="" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034132 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034137 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034140 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034143 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034149 2576 flags.go:64] FLAG: --tls-min-version="" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034151 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034154 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034157 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 30 08:11:46.042469 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034160 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034163 2576 flags.go:64] FLAG: --v="2" Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034167 2576 flags.go:64] FLAG: --version="false" Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034171 2576 flags.go:64] FLAG: --vmodule="" Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034175 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.034178 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034314 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034318 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034321 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034324 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034327 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034330 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034333 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034335 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034338 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034340 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034343 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034345 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034348 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034351 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034354 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 08:11:46.043069 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034356 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034359 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034363 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034385 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034388 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034391 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034394 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034398 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034402 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034406 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034409 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034412 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034415 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034418 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034420 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034423 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034426 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034429 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034432 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 08:11:46.043605 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034435 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034438 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034440 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034443 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034445 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034448 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034450 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034453 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034455 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034458 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034460 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034463 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034466 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034468 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034471 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034475 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034477 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034480 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034483 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 08:11:46.044070 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034486 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034490 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034493 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034495 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034499 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034502 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034505 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034508 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034510 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034512 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034515 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034518 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034520 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034522 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034525 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034527 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034530 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034533 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034535 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034538 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 08:11:46.044572 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034540 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034543 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034546 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034548 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034551 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034553 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034556 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034559 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034562 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034565 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034567 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034570 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034573 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034577 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034580 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034582 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034585 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034588 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034590 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034593 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 08:11:46.045059 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034596 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034600 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034603 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034606 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034609 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.034612 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.035245 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.042213 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.042228 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.042246 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.042252 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042315 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042320 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042323 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 08:11:46.045602 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042326 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042329 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042331 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042334 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042337 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042339 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042342 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042344 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042347 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042355 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042358 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042361 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042363 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042382 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042385 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042389 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042393 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042396 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042400 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042402 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 08:11:46.045953 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042406 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042408 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042411 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042414 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042418 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042427 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042431 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042435 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042438 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042441 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042444 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042448 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042451 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042454 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042457 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042459 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042462 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042465 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 08:11:46.046505 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042467 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042470 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042473 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042476 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042484 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042487 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042490 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042492 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042495 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042498 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042500 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042502 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042505 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042507 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042510 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042513 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042515 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042518 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042521 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042523 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 08:11:46.046950 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042526 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042529 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042531 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042534 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042536 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042539 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042542 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042544 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042547 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042549 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042552 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042554 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042557 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042559 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042562 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042564 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042567 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042576 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042579 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042581 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 08:11:46.047501 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042584 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042587 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042589 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042592 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042595 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042597 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042599 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042602 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042605 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042608 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042611 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042613 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042616 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042619 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042621 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042624 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042626 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 08:11:46.048010 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042629 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.042634 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042789 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042794 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042797 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042799 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042802 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042805 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042807 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042809 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042812 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042815 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042817 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042825 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042828 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 08:11:46.048450 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042831 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042833 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042836 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042838 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042841 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042843 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042846 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042848 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042850 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042853 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042855 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042858 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042861 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042863 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042866 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042868 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042871 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042873 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042877 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 08:11:46.048831 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042882 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042886 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042889 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042892 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042894 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042897 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042900 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042902 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042905 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042907 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042910 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042913 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042921 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042924 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042926 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042929 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042931 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042934 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042936 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042938 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 08:11:46.049298 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042941 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042943 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042946 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042948 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042951 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042953 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042956 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042959 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042962 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042964 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042967 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042969 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042971 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042974 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042976 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042978 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042981 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042983 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042986 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042988 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 08:11:46.049809 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042990 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042993 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042996 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.042999 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043002 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043011 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043014 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043017 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043020 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043022 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043025 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043028 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043030 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043033 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043035 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043038 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043040 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043043 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043045 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 08:11:46.050306 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043048 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043050 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043053 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043056 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043058 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043061 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043063 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:46.043065 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.043070 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.043184 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.045255 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.046630 2576 server.go:1019] "Starting client certificate rotation" Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.046736 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 08:11:46.051208 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.046771 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 08:11:46.051846 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.051791 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 08:11:46.057422 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.057400 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 08:11:46.071288 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.071270 2576 log.go:25] "Validated CRI v1 runtime API" Jul 30 08:11:46.073474 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.073458 2576 log.go:25] "Validated CRI v1 image API" Jul 30 08:11:46.075475 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.075453 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 30 08:11:46.077719 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.077680 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 afbf9d68-f301-4bf0-a9cb-086f2d75b163:/dev/nvme0n1p4 bc18b7f5-ac26-4fbc-a316-aef70f9d6343:/dev/nvme0n1p3] Jul 30 08:11:46.077807 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.077718 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 30 08:11:46.077860 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.077842 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 08:11:46.082789 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.082682 2576 manager.go:216] Machine: {Timestamp:2026-07-30 08:11:46.081386261 +0000 UTC m=+0.341332589 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108987 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27f0f7de13f89744baa719e39a8438 SystemUUID:ec27f0f7-de13-f897-44ba-a719e39a8438 BootID:ecd97e68-0d4b-4dc1-bf99-9bea72fedb96 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:65:c7:f7:57:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:65:c7:f7:57:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:c3:1d:f6:d1:9a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 30 08:11:46.082789 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.082783 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 30 08:11:46.082902 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.082861 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 30 08:11:46.084027 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.083995 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 30 08:11:46.084173 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.084029 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-119.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 30 08:11:46.084857 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.084846 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 30 08:11:46.084893 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.084859 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 30 08:11:46.084893 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.084867 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 08:11:46.086126 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.086115 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 08:11:46.086882 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.086872 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 30 08:11:46.088970 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.088958 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 30 08:11:46.089125 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.089115 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 30 08:11:46.089163 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.089154 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 30 08:11:46.089203 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.089193 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 30 08:11:46.089238 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.089217 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 30 08:11:46.089266 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.089238 2576 kubelet.go:398] "Adding apiserver pod source" Jul 30 08:11:46.089266 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.089251 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 30 08:11:46.089766 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.089754 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 08:11:46.089807 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.089776 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 08:11:46.093119 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.093097 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 30 08:11:46.096063 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.096046 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 30 08:11:46.096154 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.096077 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 30 08:11:46.097582 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097565 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 30 08:11:46.097582 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097583 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 30 08:11:46.097687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097590 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 30 08:11:46.097687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097597 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 30 08:11:46.097687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097609 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 30 08:11:46.097687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097617 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 30 08:11:46.097687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097626 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 30 08:11:46.097687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097631 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 30 08:11:46.097687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097640 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 30 08:11:46.097687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097645 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 30 08:11:46.097687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097655 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 30 08:11:46.097687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.097660 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 30 08:11:46.098306 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.098287 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 30 08:11:46.098306 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.098301 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 30 08:11:46.098963 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.098942 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-119.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 30 08:11:46.098963 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.098950 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 30 08:11:46.099241 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.099226 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tjjfs" Jul 30 08:11:46.100279 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.100261 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-119.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 30 08:11:46.101463 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.101449 2576 server.go:1268] "Started kubelet" Jul 30 08:11:46.101541 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.101514 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 30 08:11:46.101602 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.101529 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 30 08:11:46.101602 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.101590 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 30 08:11:46.102388 ip-10-0-136-119 systemd[1]: Started Kubernetes Kubelet. Jul 30 08:11:46.103233 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.103109 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 30 08:11:46.103741 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.103728 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 30 08:11:46.106642 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.106619 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tjjfs" Jul 30 08:11:46.107287 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.107268 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 30 08:11:46.107762 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.106965 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-119.ec2.internal.18c703070d7988f5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-119.ec2.internal,UID:ip-10-0-136-119.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-119.ec2.internal,},FirstTimestamp:2026-07-30 08:11:46.101414133 +0000 UTC m=+0.361360460,LastTimestamp:2026-07-30 08:11:46.101414133 +0000 UTC m=+0.361360460,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-119.ec2.internal,}" Jul 30 08:11:46.112599 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.112582 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 30 08:11:46.112599 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.112592 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 30 08:11:46.112722 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.112700 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:46.113453 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113437 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 30 08:11:46.113543 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113509 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 30 08:11:46.113543 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113481 2576 factory.go:55] Registering systemd factory Jul 30 08:11:46.113649 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113553 2576 factory.go:223] Registration of the systemd container factory successfully Jul 30 08:11:46.113649 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113456 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 30 08:11:46.113649 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113612 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 30 08:11:46.113649 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113629 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 30 08:11:46.113827 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113739 2576 factory.go:153] Registering CRI-O factory Jul 30 08:11:46.113827 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113758 2576 factory.go:223] Registration of the crio container factory successfully Jul 30 08:11:46.113827 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113804 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 30 08:11:46.113827 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113822 2576 factory.go:103] Registering Raw factory Jul 30 08:11:46.113988 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.113833 2576 manager.go:1196] Started watching for new ooms in manager Jul 30 08:11:46.115230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.115178 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 08:11:46.115690 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.115671 2576 manager.go:349] Starting recovery of all containers Jul 30 08:11:46.122195 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.122169 2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-119.ec2.internal\" not found" node="ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.128534 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.128518 2576 manager.go:354] Recovery completed Jul 30 08:11:46.132598 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.132586 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 08:11:46.135317 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.135301 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientMemory" Jul 30 08:11:46.135412 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.135330 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 08:11:46.135412 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.135340 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientPID" Jul 30 08:11:46.135876 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.135859 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 30 08:11:46.135876 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.135875 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 08:11:46.135991 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.135875 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 30 08:11:46.135991 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.135957 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 30 08:11:46.136754 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.136740 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientMemory" Jul 30 08:11:46.136815 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.136765 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 08:11:46.136815 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.136776 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientPID" Jul 30 08:11:46.137942 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.137930 2576 policy_none.go:49] "None policy: Start" Jul 30 08:11:46.137988 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.137946 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 30 08:11:46.137988 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.137957 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 30 08:11:46.139184 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.139173 2576 policy_none.go:47] "Start" Jul 30 08:11:46.174157 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.173966 2576 manager.go:337] "Starting Device Plugin manager" Jul 30 08:11:46.174248 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.174182 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 30 08:11:46.174248 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.174195 2576 server.go:87] "Starting device plugin registration server" Jul 30 08:11:46.174434 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.174421 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 30 08:11:46.174488 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.174435 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 30 08:11:46.174725 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.174574 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 30 08:11:46.174725 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.174657 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 30 08:11:46.174725 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.174666 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 30 08:11:46.175267 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.175247 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 30 08:11:46.175342 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.175307 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:46.275087 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.275024 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 08:11:46.276837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.276043 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientMemory" Jul 30 08:11:46.276837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.276071 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 08:11:46.276837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.276085 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientPID" Jul 30 08:11:46.276837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.276115 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.282164 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.282146 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.282250 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.282172 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-119.ec2.internal\": node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:46.284297 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.284259 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 30 08:11:46.285560 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.285542 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 30 08:11:46.285651 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.285577 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 30 08:11:46.285651 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.285598 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 30 08:11:46.285745 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.285657 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 30 08:11:46.287218 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.287203 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 08:11:46.294235 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.294208 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:46.387301 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.387257 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-119.ec2.internal"] Jul 30 08:11:46.387465 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.387352 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 08:11:46.388386 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.388355 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientMemory" Jul 30 08:11:46.388458 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.388387 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 08:11:46.388458 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.388412 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientPID" Jul 30 08:11:46.389798 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.389784 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 08:11:46.389916 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.389900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.389951 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.389940 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 08:11:46.390546 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.390528 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientMemory" Jul 30 08:11:46.390626 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.390530 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientMemory" Jul 30 08:11:46.390626 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.390581 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 08:11:46.390626 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.390594 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientPID" Jul 30 08:11:46.390626 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.390558 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 08:11:46.390760 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.390633 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientPID" Jul 30 08:11:46.391998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.391985 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.392054 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.392015 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 08:11:46.392701 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.392684 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientMemory" Jul 30 08:11:46.392782 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.392715 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 08:11:46.392782 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.392730 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeHasSufficientPID" Jul 30 08:11:46.395203 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.395186 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:46.415620 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.415592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d8dffd82708cc682767862a7cdc26a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal\" (UID: \"c5d8dffd82708cc682767862a7cdc26a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.415726 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.415624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f87618faf1b1c298cf86a88886360a92-config\") pod \"kube-apiserver-proxy-ip-10-0-136-119.ec2.internal\" (UID: \"f87618faf1b1c298cf86a88886360a92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.415726 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.415652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5d8dffd82708cc682767862a7cdc26a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal\" (UID: \"c5d8dffd82708cc682767862a7cdc26a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.421273 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.421255 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-119.ec2.internal\" not found" node="ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.426201 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.426137 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-119.ec2.internal\" not found" node="ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.495530 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.495481 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:46.516778 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.516748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5d8dffd82708cc682767862a7cdc26a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal\" (UID: \"c5d8dffd82708cc682767862a7cdc26a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.516903 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.516786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d8dffd82708cc682767862a7cdc26a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal\" (UID: \"c5d8dffd82708cc682767862a7cdc26a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.516903 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.516806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f87618faf1b1c298cf86a88886360a92-config\") pod \"kube-apiserver-proxy-ip-10-0-136-119.ec2.internal\" (UID: \"f87618faf1b1c298cf86a88886360a92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.516903 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.516866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5d8dffd82708cc682767862a7cdc26a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal\" (UID: \"c5d8dffd82708cc682767862a7cdc26a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.516998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.516895 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f87618faf1b1c298cf86a88886360a92-config\") pod \"kube-apiserver-proxy-ip-10-0-136-119.ec2.internal\" (UID: \"f87618faf1b1c298cf86a88886360a92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.516998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.516869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5d8dffd82708cc682767862a7cdc26a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal\" (UID: \"c5d8dffd82708cc682767862a7cdc26a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.596392 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.596317 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:46.698647 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.698603 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:46.724381 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.724353 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.729437 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:46.729419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-119.ec2.internal" Jul 30 08:11:46.799649 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.799600 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:46.901759 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:46.901691 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:47.003819 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:47.003788 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:47.046079 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.046040 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 30 08:11:47.046545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.046197 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 08:11:47.046545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.046201 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 08:11:47.104385 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:47.104339 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:47.111565 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.111524 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 08:06:46 +0000 UTC" deadline="2028-01-18 16:45:36.115796411 +0000 UTC" Jul 30 08:11:47.111628 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.111565 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12896h33m49.004233606s" Jul 30 08:11:47.204614 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:47.204549 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:47.232685 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.232663 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 08:11:47.305228 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:47.305196 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:47.318311 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:47.318267 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87618faf1b1c298cf86a88886360a92.slice/crio-4fa56ec36649026ec15740ecf0947633625b9c1f64f99f26ff3254abc3ceb0a0 WatchSource:0}: Error finding container 4fa56ec36649026ec15740ecf0947633625b9c1f64f99f26ff3254abc3ceb0a0: Status 404 returned error can't find the container with id 4fa56ec36649026ec15740ecf0947633625b9c1f64f99f26ff3254abc3ceb0a0 Jul 30 08:11:47.318690 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:47.318670 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d8dffd82708cc682767862a7cdc26a.slice/crio-a1c106adddd45fbfc52e1d993c7695448290064a3e9b925dcd97863f5573ff09 WatchSource:0}: Error finding container a1c106adddd45fbfc52e1d993c7695448290064a3e9b925dcd97863f5573ff09: Status 404 returned error can't find the container with id a1c106adddd45fbfc52e1d993c7695448290064a3e9b925dcd97863f5573ff09 Jul 30 08:11:47.323787 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.323773 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 08:11:47.405812 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:47.405781 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:47.506404 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:47.506324 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-119.ec2.internal\" not found" Jul 30 08:11:47.568685 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.568664 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 08:11:47.612953 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.612927 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" Jul 30 08:11:47.623843 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.623823 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 08:11:47.624771 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.624745 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-119.ec2.internal" Jul 30 08:11:47.633281 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.633264 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 08:11:47.904223 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:47.904147 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 08:11:48.018138 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.018111 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 08:11:48.090748 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.090711 2576 apiserver.go:52] "Watching apiserver" Jul 30 08:11:48.100409 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.100358 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 30 08:11:48.101731 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.101704 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-wtt8f","kube-system/konnectivity-agent-gz5j5","kube-system/kube-apiserver-proxy-ip-10-0-136-119.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2","openshift-cluster-node-tuning-operator/tuned-v6sg4","openshift-dns/node-resolver-lp4ct","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal","openshift-network-operator/iptables-alerter-lhbgs","openshift-image-registry/node-ca-s6r42","openshift-multus/multus-4zmjm","openshift-multus/multus-additional-cni-plugins-zpgxk","openshift-multus/network-metrics-daemon-rkhqs","openshift-network-diagnostics/network-check-target-xp7kt","openshift-ovn-kubernetes/ovnkube-node-nzfgq"] Jul 30 08:11:48.103492 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.103472 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.104817 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.104799 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:48.104895 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.104868 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:11:48.106102 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.106083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.106783 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.106762 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 30 08:11:48.106783 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.106772 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4rppx\"" Jul 30 08:11:48.106904 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.106808 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 30 08:11:48.106904 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.106814 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 30 08:11:48.106904 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.106760 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 30 08:11:48.110890 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.108124 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.110890 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.110612 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.111506 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.111634 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.111748 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.111776 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.111925 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mpb7f\"" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.112114 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.112149 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6v425\"" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.112424 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.112547 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.112616 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.112670 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lp4ct" Jul 30 08:11:48.112992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.112878 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 30 08:11:48.113906 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.113214 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 30 08:11:48.113906 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.113493 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 30 08:11:48.113906 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.113530 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 30 08:11:48.113906 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.113846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s6r42" Jul 30 08:11:48.114397 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.114354 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p8ggn\"" Jul 30 08:11:48.114804 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.114782 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 30 08:11:48.114804 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.114799 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 30 08:11:48.115039 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.115025 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f4sfh\"" Jul 30 08:11:48.115797 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.115721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lhbgs" Jul 30 08:11:48.115923 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.115895 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hdrhq\"" Jul 30 08:11:48.116142 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.116087 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 30 08:11:48.116142 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.116112 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 30 08:11:48.116401 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.116384 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 30 08:11:48.117103 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.117085 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:11:48.118337 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.118319 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h6zzg\"" Jul 30 08:11:48.118337 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.118331 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 30 08:11:48.118499 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.118347 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 30 08:11:48.118499 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.118455 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 30 08:11:48.118680 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.118662 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:48.118767 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.118743 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:11:48.119329 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.119313 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sr6jm\"" Jul 30 08:11:48.119415 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.119390 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 30 08:11:48.119529 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.119510 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 30 08:11:48.120355 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.120280 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.121706 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.121689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:48.121808 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.121740 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:11:48.122367 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.122350 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 30 08:11:48.122556 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.122540 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 30 08:11:48.122631 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.122594 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2cpw9\"" Jul 30 08:11:48.126626 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmg2n\" (UniqueName: \"kubernetes.io/projected/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-kube-api-access-wmg2n\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.126727 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zdbn\" (UniqueName: \"kubernetes.io/projected/bd9d1bb7-c615-40e0-b53a-d2ca26386d62-kube-api-access-9zdbn\") pod \"node-resolver-lp4ct\" (UID: \"bd9d1bb7-c615-40e0-b53a-d2ca26386d62\") " pod="openshift-dns/node-resolver-lp4ct" Jul 30 08:11:48.126727 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f5ed960c-21db-4b02-a09a-122c1c90362e-env-overrides\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.126727 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-systemd\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.126849 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d56f200-9b41-41c5-b48f-318c7d379588-multus-daemon-config\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.126849 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjwx6\" (UniqueName: \"kubernetes.io/projected/ab4c1b36-d20a-4ac3-96f3-fcd94d257061-kube-api-access-fjwx6\") pod \"node-ca-s6r42\" (UID: \"ab4c1b36-d20a-4ac3-96f3-fcd94d257061\") " pod="openshift-image-registry/node-ca-s6r42" Jul 30 08:11:48.126849 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-device-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.126993 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126859 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-tmp\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.126993 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab4c1b36-d20a-4ac3-96f3-fcd94d257061-host\") pod \"node-ca-s6r42\" (UID: \"ab4c1b36-d20a-4ac3-96f3-fcd94d257061\") " pod="openshift-image-registry/node-ca-s6r42" Jul 30 08:11:48.126993 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-systemd-units\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.126993 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-sys-fs\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.126993 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-run-systemd\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.127235 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.126996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-node-log\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.127235 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-cni-netd\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.127235 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-sysconfig\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.127235 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-run\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.127235 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-cnibin\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.127235 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127150 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-run-multus-certs\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.127235 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-run-ovn-kubernetes\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.127235 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127209 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b832196-8be8-4c5f-aaf8-d81d73659e8b-host-slash\") pod \"iptables-alerter-lhbgs\" (UID: \"6b832196-8be8-4c5f-aaf8-d81d73659e8b\") " pod="openshift-network-operator/iptables-alerter-lhbgs" Jul 30 08:11:48.127235 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-multus-cni-dir\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-sysctl-conf\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-run-netns\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127445 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-etc-openvswitch\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-socket-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b832196-8be8-4c5f-aaf8-d81d73659e8b-iptables-alerter-script\") pod \"iptables-alerter-lhbgs\" (UID: \"6b832196-8be8-4c5f-aaf8-d81d73659e8b\") " pod="openshift-network-operator/iptables-alerter-lhbgs" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-var-lib-cni-multus\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-os-release\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127530 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxwk\" (UniqueName: \"kubernetes.io/projected/f5ed960c-21db-4b02-a09a-122c1c90362e-kube-api-access-rfxwk\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-var-lib-kubelet\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab4c1b36-d20a-4ac3-96f3-fcd94d257061-serviceca\") pod \"node-ca-s6r42\" (UID: \"ab4c1b36-d20a-4ac3-96f3-fcd94d257061\") " pod="openshift-image-registry/node-ca-s6r42" Jul 30 08:11:48.127861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.127614 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-run-netns\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.128456 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-var-lib-cni-bin\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.128456 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128210 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-run-k8s-cni-cncf-io\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.128456 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-run-openvswitch\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.128456 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f5ed960c-21db-4b02-a09a-122c1c90362e-ovnkube-config\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.128456 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-modprobe-d\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.128456 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-kubernetes\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.128456 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd9d1bb7-c615-40e0-b53a-d2ca26386d62-hosts-file\") pod \"node-resolver-lp4ct\" (UID: \"bd9d1bb7-c615-40e0-b53a-d2ca26386d62\") " pod="openshift-dns/node-resolver-lp4ct" Jul 30 08:11:48.128456 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-etc-selinux\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.128785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-etc-kubernetes\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.128785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-lib-modules\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.128785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-var-lib-kubelet\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.128785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-tuned\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.128785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd9d1bb7-c615-40e0-b53a-d2ca26386d62-tmp-dir\") pod \"node-resolver-lp4ct\" (UID: \"bd9d1bb7-c615-40e0-b53a-d2ca26386d62\") " pod="openshift-dns/node-resolver-lp4ct" Jul 30 08:11:48.128785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh\") pod \"network-check-target-xp7kt\" (UID: \"b8c84511-af0e-41e8-bbaf-0cb318a69dce\") " pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:48.128785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmgdz\" (UniqueName: \"kubernetes.io/projected/663a1b86-8d54-498f-b4eb-0d0208472601-kube-api-access-wmgdz\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.128785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d56f200-9b41-41c5-b48f-318c7d379588-cni-binary-copy\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.128785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-hostroot\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.128785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-kubelet\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.128785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-log-socket\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f5ed960c-21db-4b02-a09a-122c1c90362e-ovn-node-metrics-cert\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-registration-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-sys\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-system-cni-dir\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-multus-socket-dir-parent\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-multus-conf-dir\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-sysctl-d\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-slash\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.128991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzf7h\" (UniqueName: \"kubernetes.io/projected/6d56f200-9b41-41c5-b48f-318c7d379588-kube-api-access-nzf7h\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.129015 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-host\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.129042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-var-lib-openvswitch\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.129067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f5ed960c-21db-4b02-a09a-122c1c90362e-ovnkube-script-lib\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.129091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-run-ovn\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.129114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-cni-bin\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.129138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.129249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.129161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7srj\" (UniqueName: \"kubernetes.io/projected/6b832196-8be8-4c5f-aaf8-d81d73659e8b-kube-api-access-w7srj\") pod \"iptables-alerter-lhbgs\" (UID: \"6b832196-8be8-4c5f-aaf8-d81d73659e8b\") " pod="openshift-network-operator/iptables-alerter-lhbgs" Jul 30 08:11:48.129924 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.129844 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 08:11:48.147049 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.147023 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fqkwb" Jul 30 08:11:48.157458 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.157432 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fqkwb" Jul 30 08:11:48.215036 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.215011 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 30 08:11:48.229749 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.229724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-cnibin\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.229908 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.229755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-run-multus-certs\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.229908 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.229788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-run-ovn-kubernetes\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.229908 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.229876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-run-multus-certs\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230045 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.229931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b832196-8be8-4c5f-aaf8-d81d73659e8b-host-slash\") pod \"iptables-alerter-lhbgs\" (UID: \"6b832196-8be8-4c5f-aaf8-d81d73659e8b\") " pod="openshift-network-operator/iptables-alerter-lhbgs" Jul 30 08:11:48.230045 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.229948 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-cnibin\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230045 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.229965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:48.230045 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.229993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d1a8039a-aa83-4ab0-90a3-41d6ad89f288-agent-certs\") pod \"konnectivity-agent-gz5j5\" (UID: \"d1a8039a-aa83-4ab0-90a3-41d6ad89f288\") " pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:11:48.230045 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.229993 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-run-ovn-kubernetes\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.230045 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230029 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b832196-8be8-4c5f-aaf8-d81d73659e8b-host-slash\") pod \"iptables-alerter-lhbgs\" (UID: \"6b832196-8be8-4c5f-aaf8-d81d73659e8b\") " pod="openshift-network-operator/iptables-alerter-lhbgs" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-multus-cni-dir\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-sysctl-conf\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfmpn\" (UniqueName: \"kubernetes.io/projected/f661b99d-6531-403f-8622-dec42ada1174-kube-api-access-cfmpn\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-multus-cni-dir\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-run-netns\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-etc-openvswitch\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-sysctl-conf\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-socket-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230315 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-run-netns\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230321 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-etc-openvswitch\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.230344 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230345 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b832196-8be8-4c5f-aaf8-d81d73659e8b-iptables-alerter-script\") pod \"iptables-alerter-lhbgs\" (UID: \"6b832196-8be8-4c5f-aaf8-d81d73659e8b\") " pod="openshift-network-operator/iptables-alerter-lhbgs" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-var-lib-cni-multus\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-os-release\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230424 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-socket-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxwk\" (UniqueName: \"kubernetes.io/projected/f5ed960c-21db-4b02-a09a-122c1c90362e-kube-api-access-rfxwk\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-os-release\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-var-lib-kubelet\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab4c1b36-d20a-4ac3-96f3-fcd94d257061-serviceca\") pod \"node-ca-s6r42\" (UID: \"ab4c1b36-d20a-4ac3-96f3-fcd94d257061\") " pod="openshift-image-registry/node-ca-s6r42" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-run-netns\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230574 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-var-lib-cni-multus\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-var-lib-cni-bin\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-var-lib-kubelet\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-run-k8s-cni-cncf-io\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-var-lib-cni-bin\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-host-run-k8s-cni-cncf-io\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230653 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-run-netns\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-run-openvswitch\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f5ed960c-21db-4b02-a09a-122c1c90362e-ovnkube-config\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.230998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-modprobe-d\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-run-openvswitch\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230756 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-kubernetes\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd9d1bb7-c615-40e0-b53a-d2ca26386d62-hosts-file\") pod \"node-resolver-lp4ct\" (UID: \"bd9d1bb7-c615-40e0-b53a-d2ca26386d62\") " pod="openshift-dns/node-resolver-lp4ct" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-kubernetes\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-etc-selinux\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x92lh\" (UniqueName: \"kubernetes.io/projected/67e8b7f3-fb6e-4551-ae93-189127b4eea7-kube-api-access-x92lh\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230852 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-modprobe-d\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-etc-kubernetes\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230846 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd9d1bb7-c615-40e0-b53a-d2ca26386d62-hosts-file\") pod \"node-resolver-lp4ct\" (UID: \"bd9d1bb7-c615-40e0-b53a-d2ca26386d62\") " pod="openshift-dns/node-resolver-lp4ct" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-lib-modules\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-var-lib-kubelet\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230935 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-etc-selinux\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230943 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-etc-kubernetes\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-tuned\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230968 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b832196-8be8-4c5f-aaf8-d81d73659e8b-iptables-alerter-script\") pod \"iptables-alerter-lhbgs\" (UID: \"6b832196-8be8-4c5f-aaf8-d81d73659e8b\") " pod="openshift-network-operator/iptables-alerter-lhbgs" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd9d1bb7-c615-40e0-b53a-d2ca26386d62-tmp-dir\") pod \"node-resolver-lp4ct\" (UID: \"bd9d1bb7-c615-40e0-b53a-d2ca26386d62\") " pod="openshift-dns/node-resolver-lp4ct" Jul 30 08:11:48.231837 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.230987 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-var-lib-kubelet\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh\") pod \"network-check-target-xp7kt\" (UID: \"b8c84511-af0e-41e8-bbaf-0cb318a69dce\") " pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmgdz\" (UniqueName: \"kubernetes.io/projected/663a1b86-8d54-498f-b4eb-0d0208472601-kube-api-access-wmgdz\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-lib-modules\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d56f200-9b41-41c5-b48f-318c7d379588-cni-binary-copy\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231063 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab4c1b36-d20a-4ac3-96f3-fcd94d257061-serviceca\") pod \"node-ca-s6r42\" (UID: \"ab4c1b36-d20a-4ac3-96f3-fcd94d257061\") " pod="openshift-image-registry/node-ca-s6r42" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-hostroot\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-kubelet\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-log-socket\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231170 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-log-socket\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231211 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-hostroot\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-kubelet\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231252 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f5ed960c-21db-4b02-a09a-122c1c90362e-ovn-node-metrics-cert\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-registration-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-system-cni-dir\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231348 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f5ed960c-21db-4b02-a09a-122c1c90362e-ovnkube-config\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-sys\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.232448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231426 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-sys\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-system-cni-dir\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd9d1bb7-c615-40e0-b53a-d2ca26386d62-tmp-dir\") pod \"node-resolver-lp4ct\" (UID: \"bd9d1bb7-c615-40e0-b53a-d2ca26386d62\") " pod="openshift-dns/node-resolver-lp4ct" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-multus-socket-dir-parent\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-multus-conf-dir\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231475 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-registration-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231516 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-system-cni-dir\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-sysctl-d\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231515 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-multus-socket-dir-parent\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231548 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d56f200-9b41-41c5-b48f-318c7d379588-multus-conf-dir\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-slash\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzf7h\" (UniqueName: \"kubernetes.io/projected/6d56f200-9b41-41c5-b48f-318c7d379588-kube-api-access-nzf7h\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231633 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-sysctl-d\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-host\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231652 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-slash\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231686 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d56f200-9b41-41c5-b48f-318c7d379588-cni-binary-copy\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231700 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-host\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-var-lib-openvswitch\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.233230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f5ed960c-21db-4b02-a09a-122c1c90362e-ovnkube-script-lib\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d1a8039a-aa83-4ab0-90a3-41d6ad89f288-konnectivity-ca\") pod \"konnectivity-agent-gz5j5\" (UID: \"d1a8039a-aa83-4ab0-90a3-41d6ad89f288\") " pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-var-lib-openvswitch\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-run-ovn\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-cni-bin\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231846 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-run-ovn\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7srj\" (UniqueName: \"kubernetes.io/projected/6b832196-8be8-4c5f-aaf8-d81d73659e8b-kube-api-access-w7srj\") pod \"iptables-alerter-lhbgs\" (UID: \"6b832196-8be8-4c5f-aaf8-d81d73659e8b\") " pod="openshift-network-operator/iptables-alerter-lhbgs" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231888 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-cni-bin\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231892 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67e8b7f3-fb6e-4551-ae93-189127b4eea7-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67e8b7f3-fb6e-4551-ae93-189127b4eea7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.231982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmg2n\" (UniqueName: \"kubernetes.io/projected/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-kube-api-access-wmg2n\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zdbn\" (UniqueName: \"kubernetes.io/projected/bd9d1bb7-c615-40e0-b53a-d2ca26386d62-kube-api-access-9zdbn\") pod \"node-resolver-lp4ct\" (UID: \"bd9d1bb7-c615-40e0-b53a-d2ca26386d62\") " pod="openshift-dns/node-resolver-lp4ct" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f5ed960c-21db-4b02-a09a-122c1c90362e-env-overrides\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-os-release\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-systemd\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.234182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d56f200-9b41-41c5-b48f-318c7d379588-multus-daemon-config\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjwx6\" (UniqueName: \"kubernetes.io/projected/ab4c1b36-d20a-4ac3-96f3-fcd94d257061-kube-api-access-fjwx6\") pod \"node-ca-s6r42\" (UID: \"ab4c1b36-d20a-4ac3-96f3-fcd94d257061\") " pod="openshift-image-registry/node-ca-s6r42" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-device-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da7e45bf-dfea-414e-8f0b-98242e6d10f5-kubelet-config\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-cnibin\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f5ed960c-21db-4b02-a09a-122c1c90362e-ovnkube-script-lib\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-tmp\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab4c1b36-d20a-4ac3-96f3-fcd94d257061-host\") pod \"node-ca-s6r42\" (UID: \"ab4c1b36-d20a-4ac3-96f3-fcd94d257061\") " pod="openshift-image-registry/node-ca-s6r42" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-systemd-units\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232298 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-systemd\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-sys-fs\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232358 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-device-dir\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67e8b7f3-fb6e-4551-ae93-189127b4eea7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-run-systemd\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-node-log\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-cni-netd\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.234936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-sysconfig\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232516 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f5ed960c-21db-4b02-a09a-122c1c90362e-env-overrides\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-run\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da7e45bf-dfea-414e-8f0b-98242e6d10f5-dbus\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-run\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-sysconfig\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232656 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/663a1b86-8d54-498f-b4eb-0d0208472601-sys-fs\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232684 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-run-systemd\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232694 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-node-log\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d56f200-9b41-41c5-b48f-318c7d379588-multus-daemon-config\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-systemd-units\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232761 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f5ed960c-21db-4b02-a09a-122c1c90362e-host-cni-netd\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.232866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab4c1b36-d20a-4ac3-96f3-fcd94d257061-host\") pod \"node-ca-s6r42\" (UID: \"ab4c1b36-d20a-4ac3-96f3-fcd94d257061\") " pod="openshift-image-registry/node-ca-s6r42" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.234886 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-etc-tuned\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.235081 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f5ed960c-21db-4b02-a09a-122c1c90362e-ovn-node-metrics-cert\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.235607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.235102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-tmp\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.241875 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.241852 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 08:11:48.241980 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.241880 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 08:11:48.241980 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.241895 2576 projected.go:196] Error preparing data for projected volume kube-api-access-sm6dh for pod openshift-network-diagnostics/network-check-target-xp7kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:11:48.241980 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.241959 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh podName:b8c84511-af0e-41e8-bbaf-0cb318a69dce nodeName:}" failed. No retries permitted until 2026-07-30 08:11:48.741937084 +0000 UTC m=+3.001883414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sm6dh" (UniqueName: "kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh") pod "network-check-target-xp7kt" (UID: "b8c84511-af0e-41e8-bbaf-0cb318a69dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:11:48.243979 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.243954 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjwx6\" (UniqueName: \"kubernetes.io/projected/ab4c1b36-d20a-4ac3-96f3-fcd94d257061-kube-api-access-fjwx6\") pod \"node-ca-s6r42\" (UID: \"ab4c1b36-d20a-4ac3-96f3-fcd94d257061\") " pod="openshift-image-registry/node-ca-s6r42" Jul 30 08:11:48.244802 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.244679 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxwk\" (UniqueName: \"kubernetes.io/projected/f5ed960c-21db-4b02-a09a-122c1c90362e-kube-api-access-rfxwk\") pod \"ovnkube-node-nzfgq\" (UID: \"f5ed960c-21db-4b02-a09a-122c1c90362e\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.244897 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.244842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zdbn\" (UniqueName: \"kubernetes.io/projected/bd9d1bb7-c615-40e0-b53a-d2ca26386d62-kube-api-access-9zdbn\") pod \"node-resolver-lp4ct\" (UID: \"bd9d1bb7-c615-40e0-b53a-d2ca26386d62\") " pod="openshift-dns/node-resolver-lp4ct" Jul 30 08:11:48.244897 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.244885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmgdz\" (UniqueName: \"kubernetes.io/projected/663a1b86-8d54-498f-b4eb-0d0208472601-kube-api-access-wmgdz\") pod \"aws-ebs-csi-driver-node-bngm2\" (UID: \"663a1b86-8d54-498f-b4eb-0d0208472601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.245129 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.245110 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7srj\" (UniqueName: \"kubernetes.io/projected/6b832196-8be8-4c5f-aaf8-d81d73659e8b-kube-api-access-w7srj\") pod \"iptables-alerter-lhbgs\" (UID: \"6b832196-8be8-4c5f-aaf8-d81d73659e8b\") " pod="openshift-network-operator/iptables-alerter-lhbgs" Jul 30 08:11:48.245772 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.245731 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmg2n\" (UniqueName: \"kubernetes.io/projected/d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6-kube-api-access-wmg2n\") pod \"tuned-v6sg4\" (UID: \"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6\") " pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.245900 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.245885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzf7h\" (UniqueName: \"kubernetes.io/projected/6d56f200-9b41-41c5-b48f-318c7d379588-kube-api-access-nzf7h\") pod \"multus-4zmjm\" (UID: \"6d56f200-9b41-41c5-b48f-318c7d379588\") " pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.292423 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.292358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-119.ec2.internal" event={"ID":"f87618faf1b1c298cf86a88886360a92","Type":"ContainerStarted","Data":"4fa56ec36649026ec15740ecf0947633625b9c1f64f99f26ff3254abc3ceb0a0"} Jul 30 08:11:48.293724 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.293687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" event={"ID":"c5d8dffd82708cc682767862a7cdc26a","Type":"ContainerStarted","Data":"a1c106adddd45fbfc52e1d993c7695448290064a3e9b925dcd97863f5573ff09"} Jul 30 08:11:48.333840 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.333800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmpn\" (UniqueName: \"kubernetes.io/projected/f661b99d-6531-403f-8622-dec42ada1174-kube-api-access-cfmpn\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:48.333840 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.333838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:48.334073 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.333873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x92lh\" (UniqueName: \"kubernetes.io/projected/67e8b7f3-fb6e-4551-ae93-189127b4eea7-kube-api-access-x92lh\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334073 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.333958 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 08:11:48.334073 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-system-cni-dir\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334073 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.334034 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret podName:da7e45bf-dfea-414e-8f0b-98242e6d10f5 nodeName:}" failed. No retries permitted until 2026-07-30 08:11:48.834015293 +0000 UTC m=+3.093961622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret") pod "global-pull-secret-syncer-wtt8f" (UID: "da7e45bf-dfea-414e-8f0b-98242e6d10f5") : object "kube-system"/"original-pull-secret" not registered Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d1a8039a-aa83-4ab0-90a3-41d6ad89f288-konnectivity-ca\") pod \"konnectivity-agent-gz5j5\" (UID: \"d1a8039a-aa83-4ab0-90a3-41d6ad89f288\") " pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67e8b7f3-fb6e-4551-ae93-189127b4eea7-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-system-cni-dir\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67e8b7f3-fb6e-4551-ae93-189127b4eea7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-os-release\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da7e45bf-dfea-414e-8f0b-98242e6d10f5-kubelet-config\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-cnibin\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67e8b7f3-fb6e-4551-ae93-189127b4eea7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334284 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-os-release\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da7e45bf-dfea-414e-8f0b-98242e6d10f5-dbus\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334326 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-cnibin\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:48.334975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334360 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da7e45bf-dfea-414e-8f0b-98242e6d10f5-kubelet-config\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:48.334975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d1a8039a-aa83-4ab0-90a3-41d6ad89f288-agent-certs\") pod \"konnectivity-agent-gz5j5\" (UID: \"d1a8039a-aa83-4ab0-90a3-41d6ad89f288\") " pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:11:48.334975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334714 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d1a8039a-aa83-4ab0-90a3-41d6ad89f288-konnectivity-ca\") pod \"konnectivity-agent-gz5j5\" (UID: \"d1a8039a-aa83-4ab0-90a3-41d6ad89f288\") " pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:11:48.334975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334715 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67e8b7f3-fb6e-4551-ae93-189127b4eea7-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334735 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da7e45bf-dfea-414e-8f0b-98242e6d10f5-dbus\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:48.334975 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.334786 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:11:48.334975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334797 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67e8b7f3-fb6e-4551-ae93-189127b4eea7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334843 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67e8b7f3-fb6e-4551-ae93-189127b4eea7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.334975 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.334850 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs podName:f661b99d-6531-403f-8622-dec42ada1174 nodeName:}" failed. No retries permitted until 2026-07-30 08:11:48.834834047 +0000 UTC m=+3.094780374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs") pod "network-metrics-daemon-rkhqs" (UID: "f661b99d-6531-403f-8622-dec42ada1174") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:11:48.334975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.334891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67e8b7f3-fb6e-4551-ae93-189127b4eea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.337073 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.337048 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d1a8039a-aa83-4ab0-90a3-41d6ad89f288-agent-certs\") pod \"konnectivity-agent-gz5j5\" (UID: \"d1a8039a-aa83-4ab0-90a3-41d6ad89f288\") " pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:11:48.344924 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.344899 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x92lh\" (UniqueName: \"kubernetes.io/projected/67e8b7f3-fb6e-4551-ae93-189127b4eea7-kube-api-access-x92lh\") pod \"multus-additional-cni-plugins-zpgxk\" (UID: \"67e8b7f3-fb6e-4551-ae93-189127b4eea7\") " pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.344924 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.344916 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmpn\" (UniqueName: \"kubernetes.io/projected/f661b99d-6531-403f-8622-dec42ada1174-kube-api-access-cfmpn\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:48.415986 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.415911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4zmjm" Jul 30 08:11:48.423782 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.423761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:11:48.432415 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.432391 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" Jul 30 08:11:48.437109 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.437088 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" Jul 30 08:11:48.444717 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.444699 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lp4ct" Jul 30 08:11:48.452071 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.452044 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s6r42" Jul 30 08:11:48.459281 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.459263 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lhbgs" Jul 30 08:11:48.466077 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.466058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:11:48.471318 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.471299 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zpgxk" Jul 30 08:11:48.744026 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:48.743996 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25a97b3_406f_4e5b_a4b1_1bc1770ac7b6.slice/crio-12aa0bf5bf357b2121a3503eb2518223033cd2ef7cf3733346bc8d21eccc0349 WatchSource:0}: Error finding container 12aa0bf5bf357b2121a3503eb2518223033cd2ef7cf3733346bc8d21eccc0349: Status 404 returned error can't find the container with id 12aa0bf5bf357b2121a3503eb2518223033cd2ef7cf3733346bc8d21eccc0349 Jul 30 08:11:48.749749 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:48.749722 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4c1b36_d20a_4ac3_96f3_fcd94d257061.slice/crio-b145d49c192154f73dff786969d922c5550f284ea45a347ef1c22291ccd4e99d WatchSource:0}: Error finding container b145d49c192154f73dff786969d922c5550f284ea45a347ef1c22291ccd4e99d: Status 404 returned error can't find the container with id b145d49c192154f73dff786969d922c5550f284ea45a347ef1c22291ccd4e99d Jul 30 08:11:48.750204 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:48.750111 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b832196_8be8_4c5f_aaf8_d81d73659e8b.slice/crio-b86ded6ad65c14be5af5cfcfd5b1e1de76d045a387bda87f3202ce94d8954a0b WatchSource:0}: Error finding container b86ded6ad65c14be5af5cfcfd5b1e1de76d045a387bda87f3202ce94d8954a0b: Status 404 returned error can't find the container with id b86ded6ad65c14be5af5cfcfd5b1e1de76d045a387bda87f3202ce94d8954a0b Jul 30 08:11:48.750961 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:48.750935 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ed960c_21db_4b02_a09a_122c1c90362e.slice/crio-a821a3d7b25213f8bb014ac3d65f6b31b87309cba1b6dc46e11c204d8dcdcf7d WatchSource:0}: Error finding container a821a3d7b25213f8bb014ac3d65f6b31b87309cba1b6dc46e11c204d8dcdcf7d: Status 404 returned error can't find the container with id a821a3d7b25213f8bb014ac3d65f6b31b87309cba1b6dc46e11c204d8dcdcf7d Jul 30 08:11:48.751939 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:48.751850 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e8b7f3_fb6e_4551_ae93_189127b4eea7.slice/crio-4464488b5967606dfe251dd93cffb8470260f8aecc9fa20844776d1d95b04f94 WatchSource:0}: Error finding container 4464488b5967606dfe251dd93cffb8470260f8aecc9fa20844776d1d95b04f94: Status 404 returned error can't find the container with id 4464488b5967606dfe251dd93cffb8470260f8aecc9fa20844776d1d95b04f94 Jul 30 08:11:48.753044 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:48.752784 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a8039a_aa83_4ab0_90a3_41d6ad89f288.slice/crio-2ee5da159f0448bd4868c4ec999fd68ead5103451e783a6641c3982e74eac139 WatchSource:0}: Error finding container 2ee5da159f0448bd4868c4ec999fd68ead5103451e783a6641c3982e74eac139: Status 404 returned error can't find the container with id 2ee5da159f0448bd4868c4ec999fd68ead5103451e783a6641c3982e74eac139 Jul 30 08:11:48.754198 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:48.754158 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9d1bb7_c615_40e0_b53a_d2ca26386d62.slice/crio-bb580fbe155e96d6292cc02be6fbbc954a5d25c20936cad6b6e539984af06686 WatchSource:0}: Error finding container bb580fbe155e96d6292cc02be6fbbc954a5d25c20936cad6b6e539984af06686: Status 404 returned error can't find the container with id bb580fbe155e96d6292cc02be6fbbc954a5d25c20936cad6b6e539984af06686 Jul 30 08:11:48.756551 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:48.756527 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d56f200_9b41_41c5_b48f_318c7d379588.slice/crio-aa93c504a2053cebad72f55684f6707709a78ad9bc6a3256d4fe6dbce6bbe1cd WatchSource:0}: Error finding container aa93c504a2053cebad72f55684f6707709a78ad9bc6a3256d4fe6dbce6bbe1cd: Status 404 returned error can't find the container with id aa93c504a2053cebad72f55684f6707709a78ad9bc6a3256d4fe6dbce6bbe1cd Jul 30 08:11:48.756933 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:11:48.756824 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663a1b86_8d54_498f_b4eb_0d0208472601.slice/crio-c8e891c3f1f1279ebf482f37cd966f908394f5994bfd7954a89fabaf137f7677 WatchSource:0}: Error finding container c8e891c3f1f1279ebf482f37cd966f908394f5994bfd7954a89fabaf137f7677: Status 404 returned error can't find the container with id c8e891c3f1f1279ebf482f37cd966f908394f5994bfd7954a89fabaf137f7677 Jul 30 08:11:48.837193 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.837171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:48.837293 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.837201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:48.837293 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:48.837227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh\") pod \"network-check-target-xp7kt\" (UID: \"b8c84511-af0e-41e8-bbaf-0cb318a69dce\") " pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:48.837368 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.837309 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 08:11:48.837368 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.837319 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 08:11:48.837368 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.837331 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 08:11:48.837368 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.837340 2576 projected.go:196] Error preparing data for projected volume kube-api-access-sm6dh for pod openshift-network-diagnostics/network-check-target-xp7kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:11:48.837368 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.837362 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret podName:da7e45bf-dfea-414e-8f0b-98242e6d10f5 nodeName:}" failed. No retries permitted until 2026-07-30 08:11:49.837347247 +0000 UTC m=+4.097293561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret") pod "global-pull-secret-syncer-wtt8f" (UID: "da7e45bf-dfea-414e-8f0b-98242e6d10f5") : object "kube-system"/"original-pull-secret" not registered Jul 30 08:11:48.837550 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.837309 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:11:48.837550 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.837391 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh podName:b8c84511-af0e-41e8-bbaf-0cb318a69dce nodeName:}" failed. No retries permitted until 2026-07-30 08:11:49.837384093 +0000 UTC m=+4.097330407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sm6dh" (UniqueName: "kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh") pod "network-check-target-xp7kt" (UID: "b8c84511-af0e-41e8-bbaf-0cb318a69dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:11:48.837550 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:48.837427 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs podName:f661b99d-6531-403f-8622-dec42ada1174 nodeName:}" failed. No retries permitted until 2026-07-30 08:11:49.8374156 +0000 UTC m=+4.097361918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs") pod "network-metrics-daemon-rkhqs" (UID: "f661b99d-6531-403f-8622-dec42ada1174") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:11:49.159810 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.159525 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 08:06:48 +0000 UTC" deadline="2028-02-21 19:04:47.075586619 +0000 UTC" Jul 30 08:11:49.159810 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.159565 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13714h52m57.916026909s" Jul 30 08:11:49.286889 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.286353 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:49.286889 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:49.286506 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:11:49.300862 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.300729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" event={"ID":"f5ed960c-21db-4b02-a09a-122c1c90362e","Type":"ContainerStarted","Data":"a821a3d7b25213f8bb014ac3d65f6b31b87309cba1b6dc46e11c204d8dcdcf7d"} Jul 30 08:11:49.303268 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.303186 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4zmjm" event={"ID":"6d56f200-9b41-41c5-b48f-318c7d379588","Type":"ContainerStarted","Data":"aa93c504a2053cebad72f55684f6707709a78ad9bc6a3256d4fe6dbce6bbe1cd"} Jul 30 08:11:49.306588 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.306523 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lp4ct" event={"ID":"bd9d1bb7-c615-40e0-b53a-d2ca26386d62","Type":"ContainerStarted","Data":"bb580fbe155e96d6292cc02be6fbbc954a5d25c20936cad6b6e539984af06686"} Jul 30 08:11:49.313750 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.313715 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lhbgs" event={"ID":"6b832196-8be8-4c5f-aaf8-d81d73659e8b","Type":"ContainerStarted","Data":"b86ded6ad65c14be5af5cfcfd5b1e1de76d045a387bda87f3202ce94d8954a0b"} Jul 30 08:11:49.319522 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.319491 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s6r42" event={"ID":"ab4c1b36-d20a-4ac3-96f3-fcd94d257061","Type":"ContainerStarted","Data":"b145d49c192154f73dff786969d922c5550f284ea45a347ef1c22291ccd4e99d"} Jul 30 08:11:49.324430 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.324398 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" event={"ID":"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6","Type":"ContainerStarted","Data":"12aa0bf5bf357b2121a3503eb2518223033cd2ef7cf3733346bc8d21eccc0349"} Jul 30 08:11:49.329723 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.329692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-119.ec2.internal" event={"ID":"f87618faf1b1c298cf86a88886360a92","Type":"ContainerStarted","Data":"42db25fc1d58844a96415d61b3133d033d87033a47b9297f42339e4926c5e8aa"} Jul 30 08:11:49.349746 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.349704 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" event={"ID":"663a1b86-8d54-498f-b4eb-0d0208472601","Type":"ContainerStarted","Data":"c8e891c3f1f1279ebf482f37cd966f908394f5994bfd7954a89fabaf137f7677"} Jul 30 08:11:49.360558 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.360520 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gz5j5" event={"ID":"d1a8039a-aa83-4ab0-90a3-41d6ad89f288","Type":"ContainerStarted","Data":"2ee5da159f0448bd4868c4ec999fd68ead5103451e783a6641c3982e74eac139"} Jul 30 08:11:49.367952 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.367915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpgxk" event={"ID":"67e8b7f3-fb6e-4551-ae93-189127b4eea7","Type":"ContainerStarted","Data":"4464488b5967606dfe251dd93cffb8470260f8aecc9fa20844776d1d95b04f94"} Jul 30 08:11:49.844957 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.844922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh\") pod \"network-check-target-xp7kt\" (UID: \"b8c84511-af0e-41e8-bbaf-0cb318a69dce\") " pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:49.845154 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.844978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:49.845154 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:49.844997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:49.845154 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:49.845093 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 08:11:49.845318 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:49.845157 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret podName:da7e45bf-dfea-414e-8f0b-98242e6d10f5 nodeName:}" failed. No retries permitted until 2026-07-30 08:11:51.845139008 +0000 UTC m=+6.105085326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret") pod "global-pull-secret-syncer-wtt8f" (UID: "da7e45bf-dfea-414e-8f0b-98242e6d10f5") : object "kube-system"/"original-pull-secret" not registered Jul 30 08:11:49.845639 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:49.845591 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 08:11:49.845639 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:49.845610 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 08:11:49.845639 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:49.845620 2576 projected.go:196] Error preparing data for projected volume kube-api-access-sm6dh for pod openshift-network-diagnostics/network-check-target-xp7kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:11:49.845836 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:49.845655 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh podName:b8c84511-af0e-41e8-bbaf-0cb318a69dce nodeName:}" failed. No retries permitted until 2026-07-30 08:11:51.845644865 +0000 UTC m=+6.105591179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sm6dh" (UniqueName: "kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh") pod "network-check-target-xp7kt" (UID: "b8c84511-af0e-41e8-bbaf-0cb318a69dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:11:49.845836 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:49.845699 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:11:49.845836 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:49.845754 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs podName:f661b99d-6531-403f-8622-dec42ada1174 nodeName:}" failed. No retries permitted until 2026-07-30 08:11:51.845714899 +0000 UTC m=+6.105661213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs") pod "network-metrics-daemon-rkhqs" (UID: "f661b99d-6531-403f-8622-dec42ada1174") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:11:50.160806 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:50.160460 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 08:06:48 +0000 UTC" deadline="2028-01-16 03:38:57.366086237 +0000 UTC" Jul 30 08:11:50.160806 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:50.160757 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12835h27m7.205334103s" Jul 30 08:11:50.289094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:50.288186 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:50.289094 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:50.288330 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:11:50.289094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:50.288736 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:50.289094 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:50.288834 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:11:51.286817 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:51.286782 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:51.287289 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:51.286933 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:11:51.862169 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:51.862126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh\") pod \"network-check-target-xp7kt\" (UID: \"b8c84511-af0e-41e8-bbaf-0cb318a69dce\") " pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:51.862364 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:51.862208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:51.862364 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:51.862240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:51.862520 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:51.862394 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 08:11:51.862520 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:51.862462 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret podName:da7e45bf-dfea-414e-8f0b-98242e6d10f5 nodeName:}" failed. No retries permitted until 2026-07-30 08:11:55.862443987 +0000 UTC m=+10.122390306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret") pod "global-pull-secret-syncer-wtt8f" (UID: "da7e45bf-dfea-414e-8f0b-98242e6d10f5") : object "kube-system"/"original-pull-secret" not registered Jul 30 08:11:51.862907 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:51.862888 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 08:11:51.862993 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:51.862916 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 08:11:51.862993 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:51.862930 2576 projected.go:196] Error preparing data for projected volume kube-api-access-sm6dh for pod openshift-network-diagnostics/network-check-target-xp7kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:11:51.862993 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:51.862977 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh podName:b8c84511-af0e-41e8-bbaf-0cb318a69dce nodeName:}" failed. No retries permitted until 2026-07-30 08:11:55.862961596 +0000 UTC m=+10.122907916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sm6dh" (UniqueName: "kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh") pod "network-check-target-xp7kt" (UID: "b8c84511-af0e-41e8-bbaf-0cb318a69dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:11:51.863144 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:51.863039 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:11:51.863144 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:51.863071 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs podName:f661b99d-6531-403f-8622-dec42ada1174 nodeName:}" failed. No retries permitted until 2026-07-30 08:11:55.863060161 +0000 UTC m=+10.123006478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs") pod "network-metrics-daemon-rkhqs" (UID: "f661b99d-6531-403f-8622-dec42ada1174") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:11:52.286599 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:52.286057 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:52.286599 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:52.286202 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:11:52.286599 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:52.286252 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:52.286599 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:52.286394 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:11:53.286882 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:53.286403 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:53.286882 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:53.286537 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:11:54.290155 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:54.290080 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:54.290596 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:54.290287 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:11:54.290596 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:54.290402 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:54.290596 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:54.290551 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:11:55.286572 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:55.286157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:55.286572 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:55.286296 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:11:55.898985 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:55.898941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:55.899475 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:55.899000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:55.899475 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:55.899050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh\") pod \"network-check-target-xp7kt\" (UID: \"b8c84511-af0e-41e8-bbaf-0cb318a69dce\") " pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:55.899475 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:55.899209 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 08:11:55.899475 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:55.899230 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 08:11:55.899475 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:55.899244 2576 projected.go:196] Error preparing data for projected volume kube-api-access-sm6dh for pod openshift-network-diagnostics/network-check-target-xp7kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:11:55.899475 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:55.899308 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh podName:b8c84511-af0e-41e8-bbaf-0cb318a69dce nodeName:}" failed. No retries permitted until 2026-07-30 08:12:03.899290073 +0000 UTC m=+18.159236399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sm6dh" (UniqueName: "kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh") pod "network-check-target-xp7kt" (UID: "b8c84511-af0e-41e8-bbaf-0cb318a69dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:11:55.899798 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:55.899735 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:11:55.899798 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:55.899787 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs podName:f661b99d-6531-403f-8622-dec42ada1174 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:03.899772357 +0000 UTC m=+18.159718674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs") pod "network-metrics-daemon-rkhqs" (UID: "f661b99d-6531-403f-8622-dec42ada1174") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:11:55.899898 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:55.899844 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 08:11:55.899898 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:55.899878 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret podName:da7e45bf-dfea-414e-8f0b-98242e6d10f5 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:03.899866344 +0000 UTC m=+18.159812662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret") pod "global-pull-secret-syncer-wtt8f" (UID: "da7e45bf-dfea-414e-8f0b-98242e6d10f5") : object "kube-system"/"original-pull-secret" not registered Jul 30 08:11:56.287405 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:56.287314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:56.287545 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:56.287439 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:11:56.287874 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:56.287850 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:56.287992 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:56.287970 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:11:57.286400 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:57.286307 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:57.286845 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:57.286451 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:11:58.286402 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:58.286349 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:11:58.286847 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:58.286501 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:11:58.286847 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:58.286536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:11:58.286847 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:58.286598 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:11:59.286440 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:11:59.286395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:11:59.286804 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:11:59.286508 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:12:00.286463 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:00.286421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:00.286917 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:00.286464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:00.286917 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:00.286584 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:12:00.286917 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:00.286725 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:12:01.286638 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:01.286602 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:01.287041 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:01.286740 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:12:02.286660 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:02.286619 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:02.287147 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:02.286763 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:12:02.287147 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:02.286816 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:02.287147 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:02.286932 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:12:03.286168 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:03.286134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:03.286358 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:03.286262 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:12:03.957773 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:03.957734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:03.958234 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:03.957809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh\") pod \"network-check-target-xp7kt\" (UID: \"b8c84511-af0e-41e8-bbaf-0cb318a69dce\") " pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:03.958234 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:03.957866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:03.958234 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:03.957905 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 08:12:03.958234 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:03.957919 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 08:12:03.958234 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:03.957939 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 08:12:03.958234 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:03.957951 2576 projected.go:196] Error preparing data for projected volume kube-api-access-sm6dh for pod openshift-network-diagnostics/network-check-target-xp7kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:12:03.958234 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:03.957966 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:12:03.958234 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:03.957989 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret podName:da7e45bf-dfea-414e-8f0b-98242e6d10f5 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:19.957968894 +0000 UTC m=+34.217915226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret") pod "global-pull-secret-syncer-wtt8f" (UID: "da7e45bf-dfea-414e-8f0b-98242e6d10f5") : object "kube-system"/"original-pull-secret" not registered Jul 30 08:12:03.958234 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:03.958002 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh podName:b8c84511-af0e-41e8-bbaf-0cb318a69dce nodeName:}" failed. No retries permitted until 2026-07-30 08:12:19.957996724 +0000 UTC m=+34.217943038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sm6dh" (UniqueName: "kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh") pod "network-check-target-xp7kt" (UID: "b8c84511-af0e-41e8-bbaf-0cb318a69dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:12:03.958234 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:03.958013 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs podName:f661b99d-6531-403f-8622-dec42ada1174 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:19.958007437 +0000 UTC m=+34.217953751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs") pod "network-metrics-daemon-rkhqs" (UID: "f661b99d-6531-403f-8622-dec42ada1174") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:12:04.286136 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:04.286051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:04.286286 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:04.286051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:04.286286 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:04.286211 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:12:04.286286 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:04.286274 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:12:05.286193 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:05.286157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:05.286623 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:05.286291 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:12:06.287533 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:06.287489 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:06.287968 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:06.287609 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:12:06.287968 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:06.287649 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:06.287968 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:06.287720 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:12:07.286210 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:07.286178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:07.286347 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:07.286272 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:12:08.287019 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.286739 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:08.287720 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.286765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:08.287720 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:08.287160 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:12:08.287720 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:08.287254 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:12:08.417511 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.417477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s6r42" event={"ID":"ab4c1b36-d20a-4ac3-96f3-fcd94d257061","Type":"ContainerStarted","Data":"631297d40632c34801657237d498971dfcbfb59a9a0f43af10adf49ef4aa371b"} Jul 30 08:12:08.418830 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.418802 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" event={"ID":"d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6","Type":"ContainerStarted","Data":"2cdf2b062101b267fb644283fa4b98e6664db25ad5958fae96b0330045a325d1"} Jul 30 08:12:08.420179 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.420157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" event={"ID":"663a1b86-8d54-498f-b4eb-0d0208472601","Type":"ContainerStarted","Data":"fad7c33e3897e28213d3839d90cf520e75657bdb7d1e35e68a2de3d9f193dcfb"} Jul 30 08:12:08.421412 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.421387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gz5j5" event={"ID":"d1a8039a-aa83-4ab0-90a3-41d6ad89f288","Type":"ContainerStarted","Data":"245dd1cfb0e5af093ca3724bd512f1b51bedf0ef3b7004062a1cc671dad7f8bb"} Jul 30 08:12:08.422889 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.422849 2576 generic.go:356] "Generic (PLEG): container finished" podID="67e8b7f3-fb6e-4551-ae93-189127b4eea7" containerID="aa9867e79bfa1d5fa44fe24b11f5fdfa74fd8ff63eb302ed1ea3c7cc0d74ac04" exitCode=0 Jul 30 08:12:08.422972 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.422942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpgxk" event={"ID":"67e8b7f3-fb6e-4551-ae93-189127b4eea7","Type":"ContainerDied","Data":"aa9867e79bfa1d5fa44fe24b11f5fdfa74fd8ff63eb302ed1ea3c7cc0d74ac04"} Jul 30 08:12:08.425691 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.425615 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:12:08.425988 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.425960 2576 generic.go:356] "Generic (PLEG): container finished" podID="f5ed960c-21db-4b02-a09a-122c1c90362e" containerID="c86a45f6a38b261e9cea3b48b83987dd4d0d2d213697179275cc5b04b509bf0f" exitCode=1 Jul 30 08:12:08.426049 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.426022 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" event={"ID":"f5ed960c-21db-4b02-a09a-122c1c90362e","Type":"ContainerStarted","Data":"1c8e14a935d4abf7d5dbfacab93f1d9bde067fdb230d037710d998ad208eebc4"} Jul 30 08:12:08.426113 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.426095 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" event={"ID":"f5ed960c-21db-4b02-a09a-122c1c90362e","Type":"ContainerStarted","Data":"6a97b9793692a1aaa54e9bc0012951962d2876164c1bd7a97d307cff5ba1aef5"} Jul 30 08:12:08.426169 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.426157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" event={"ID":"f5ed960c-21db-4b02-a09a-122c1c90362e","Type":"ContainerStarted","Data":"7c25b507cf090980e5f3c65f3477e58cfb1cd660d58b74374666bbd643f858a0"} Jul 30 08:12:08.426222 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.426208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" event={"ID":"f5ed960c-21db-4b02-a09a-122c1c90362e","Type":"ContainerDied","Data":"c86a45f6a38b261e9cea3b48b83987dd4d0d2d213697179275cc5b04b509bf0f"} Jul 30 08:12:08.426274 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.426252 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" event={"ID":"f5ed960c-21db-4b02-a09a-122c1c90362e","Type":"ContainerStarted","Data":"af1e632361c033493ea02372c63609a54d2395f4de50b55c16c9d703ab60bc0a"} Jul 30 08:12:08.427578 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.427557 2576 generic.go:356] "Generic (PLEG): container finished" podID="c5d8dffd82708cc682767862a7cdc26a" containerID="625ddcda08a6b0ba7448a3fd633e46ea1be1d6afbfa5679998a3593f808e8bd5" exitCode=0 Jul 30 08:12:08.427666 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.427627 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" event={"ID":"c5d8dffd82708cc682767862a7cdc26a","Type":"ContainerDied","Data":"625ddcda08a6b0ba7448a3fd633e46ea1be1d6afbfa5679998a3593f808e8bd5"} Jul 30 08:12:08.428976 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.428958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4zmjm" event={"ID":"6d56f200-9b41-41c5-b48f-318c7d379588","Type":"ContainerStarted","Data":"b6d52145786a7ec0e7d0c7e545f5bf9f0286529c38fde530246cf5a530650118"} Jul 30 08:12:08.430236 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.430217 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lp4ct" event={"ID":"bd9d1bb7-c615-40e0-b53a-d2ca26386d62","Type":"ContainerStarted","Data":"6f5fd5058013c1273cddf807f182e20b9d302ccbaba0b6f917a857db342643b8"} Jul 30 08:12:08.439475 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.439426 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-119.ec2.internal" podStartSLOduration=21.439414088 podStartE2EDuration="21.439414088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:11:49.355525077 +0000 UTC m=+3.615471420" watchObservedRunningTime="2026-07-30 08:12:08.439414088 +0000 UTC m=+22.699360424" Jul 30 08:12:08.459997 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.459961 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lp4ct" podStartSLOduration=3.65959901 podStartE2EDuration="22.459948619s" totalImagesPullingTime="18.800349609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:08.459785001 +0000 UTC m=+22.719731336" watchObservedRunningTime="2026-07-30 08:12:08.459948619 +0000 UTC m=+22.719894955" Jul 30 08:12:08.460096 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.460079 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s6r42" podStartSLOduration=3.696321493 podStartE2EDuration="22.460074679s" totalImagesPullingTime="18.763753186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:08.438960848 +0000 UTC m=+22.698907185" watchObservedRunningTime="2026-07-30 08:12:08.460074679 +0000 UTC m=+22.720021015" Jul 30 08:12:08.528754 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.528577 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gz5j5" podStartSLOduration=8.230803009 podStartE2EDuration="22.528562086s" totalImagesPullingTime="14.297759077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:08.491004593 +0000 UTC m=+22.750950942" watchObservedRunningTime="2026-07-30 08:12:08.528562086 +0000 UTC m=+22.788508485" Jul 30 08:12:08.528930 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:08.528900 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v6sg4" podStartSLOduration=3.71629366 podStartE2EDuration="22.528889635s" totalImagesPullingTime="18.812595975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:08.526074984 +0000 UTC m=+22.786021321" watchObservedRunningTime="2026-07-30 08:12:08.528889635 +0000 UTC m=+22.788835970" Jul 30 08:12:09.286103 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:09.286077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:09.286223 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:09.286203 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:12:09.367952 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:09.367923 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 30 08:12:09.434731 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:09.434696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" event={"ID":"663a1b86-8d54-498f-b4eb-0d0208472601","Type":"ContainerStarted","Data":"baa4f258b650f709ff29411707f1b7b4e7e129bff9f6565c28377726add3b716"} Jul 30 08:12:09.437722 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:09.437674 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:12:09.438156 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:09.438134 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" event={"ID":"f5ed960c-21db-4b02-a09a-122c1c90362e","Type":"ContainerStarted","Data":"7235f7c7c1908dd0bb2642bb86a72f7011eb467c09badefdf8f91e30ac5a4aff"} Jul 30 08:12:09.439793 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:09.439769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lhbgs" event={"ID":"6b832196-8be8-4c5f-aaf8-d81d73659e8b","Type":"ContainerStarted","Data":"e5cf565a6c63f9ff2c9a98036455bfe9ffc4b964906951187b4d9d2fab4afc92"} Jul 30 08:12:09.458174 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:09.458136 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4zmjm" podStartSLOduration=4.203894176 podStartE2EDuration="23.45812117s" totalImagesPullingTime="19.254226994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:08.583874094 +0000 UTC m=+22.843820429" watchObservedRunningTime="2026-07-30 08:12:09.45812117 +0000 UTC m=+23.718067508" Jul 30 08:12:09.472552 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:09.472500 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lhbgs" podStartSLOduration=4.665212069 podStartE2EDuration="23.4724829s" totalImagesPullingTime="18.807270831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:09.47200455 +0000 UTC m=+23.731950888" watchObservedRunningTime="2026-07-30 08:12:09.4724829 +0000 UTC m=+23.732429241" Jul 30 08:12:10.187994 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:10.187839 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T08:12:09.3679449Z","UUID":"6e462864-3367-41f0-b7d0-8213d2b095fe","Handler":null,"Name":"","Endpoint":""} Jul 30 08:12:10.189736 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:10.189709 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 30 08:12:10.189841 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:10.189760 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 30 08:12:10.286036 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:10.286004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:10.286211 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:10.286151 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:12:10.286278 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:10.286204 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:10.286351 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:10.286321 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:12:10.443336 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:10.443273 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" event={"ID":"c5d8dffd82708cc682767862a7cdc26a","Type":"ContainerStarted","Data":"7b6bba9b190ee954c346e938f6fbcd0b8d192517768476e20dfcd19355cd0928"} Jul 30 08:12:11.286317 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:11.286280 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:11.286533 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:11.286439 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:12:11.447954 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:11.447916 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" event={"ID":"663a1b86-8d54-498f-b4eb-0d0208472601","Type":"ContainerStarted","Data":"b3b974eac17cdbdb81ff5a79422ed12cd0a7d834dda5816428b71adb5aa4f301"} Jul 30 08:12:11.451804 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:11.451776 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:12:11.452264 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:11.452242 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" event={"ID":"f5ed960c-21db-4b02-a09a-122c1c90362e","Type":"ContainerStarted","Data":"4dde2eb1ebebe33aabba5916fa9d99a23e106732e05292367e8c6fd22edd97d1"} Jul 30 08:12:11.466214 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:11.466175 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bngm2" podStartSLOduration=4.637181974 podStartE2EDuration="25.466161968s" totalImagesPullingTime="20.828979994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:11.465873804 +0000 UTC m=+25.725820174" watchObservedRunningTime="2026-07-30 08:12:11.466161968 +0000 UTC m=+25.726108303" Jul 30 08:12:11.466402 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:11.466368 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-119.ec2.internal" podStartSLOduration=24.466361683 podStartE2EDuration="24.466361683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:10.463228325 +0000 UTC m=+24.723174662" watchObservedRunningTime="2026-07-30 08:12:11.466361683 +0000 UTC m=+25.726308020" Jul 30 08:12:12.286556 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:12.286519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:12.286745 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:12.286572 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:12.286745 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:12.286659 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:12:12.286856 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:12.286782 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:12:12.951618 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:12.951194 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:12:12.952808 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:12.951770 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:12:13.286363 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.286294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:13.286495 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:13.286415 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:12:13.458863 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.458820 2576 generic.go:356] "Generic (PLEG): container finished" podID="67e8b7f3-fb6e-4551-ae93-189127b4eea7" containerID="428283e72d34106a7a9ca0569cb0e7244370c206745cc65de82728e75963fa2e" exitCode=0 Jul 30 08:12:13.459031 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.458911 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpgxk" event={"ID":"67e8b7f3-fb6e-4551-ae93-189127b4eea7","Type":"ContainerDied","Data":"428283e72d34106a7a9ca0569cb0e7244370c206745cc65de82728e75963fa2e"} Jul 30 08:12:13.462491 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.462471 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:12:13.462832 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.462814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" event={"ID":"f5ed960c-21db-4b02-a09a-122c1c90362e","Type":"ContainerStarted","Data":"d49c433b5b5c8e9743e21ee6169c43a354d4d9db33d37d566d22e88d5dacb5f9"} Jul 30 08:12:13.463090 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.463069 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:12:13.463909 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.463188 2576 scope.go:117] "RemoveContainer" containerID="c86a45f6a38b261e9cea3b48b83987dd4d0d2d213697179275cc5b04b509bf0f" Jul 30 08:12:13.463909 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.463220 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:12:13.463909 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.463261 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:12:13.463909 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.463313 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:12:13.463909 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.463632 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gz5j5" Jul 30 08:12:13.478681 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.478660 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:12:13.479017 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:13.479002 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:12:14.289237 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:14.289211 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:14.289822 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:14.289314 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:12:14.289822 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:14.289380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:14.289822 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:14.289474 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:12:14.468364 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:14.468202 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:12:14.468726 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:14.468695 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" event={"ID":"f5ed960c-21db-4b02-a09a-122c1c90362e","Type":"ContainerStarted","Data":"239f20739707ea1d9e994364955437e041db9c7c4b238dfa0632464165efb890"} Jul 30 08:12:14.512815 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:14.512766 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" podStartSLOduration=9.667568133 podStartE2EDuration="28.51275228s" totalImagesPullingTime="18.845184147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:14.511202854 +0000 UTC m=+28.771149201" watchObservedRunningTime="2026-07-30 08:12:14.51275228 +0000 UTC m=+28.772698617" Jul 30 08:12:15.145534 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:15.145506 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xp7kt"] Jul 30 08:12:15.145748 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:15.145617 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:15.145748 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:15.145718 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:12:15.146159 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:15.146137 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wtt8f"] Jul 30 08:12:15.146291 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:15.146236 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:15.146348 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:15.146327 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:12:15.146733 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:15.146712 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rkhqs"] Jul 30 08:12:15.146833 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:15.146792 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:15.146899 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:15.146880 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:12:16.475165 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:16.475097 2576 generic.go:356] "Generic (PLEG): container finished" podID="67e8b7f3-fb6e-4551-ae93-189127b4eea7" containerID="c1efdde96cf0ae71be117b590d310a4f010fb9ec2daddc7fd660f85e95368fe2" exitCode=0 Jul 30 08:12:16.475165 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:16.475157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpgxk" event={"ID":"67e8b7f3-fb6e-4551-ae93-189127b4eea7","Type":"ContainerDied","Data":"c1efdde96cf0ae71be117b590d310a4f010fb9ec2daddc7fd660f85e95368fe2"} Jul 30 08:12:17.286751 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:17.286718 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:17.286881 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:17.286719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:17.286881 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:17.286819 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:12:17.286968 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:17.286906 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:12:17.286968 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:17.286718 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:17.287030 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:17.286975 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:12:19.286955 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.286758 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:19.287306 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.286768 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:19.287306 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.287041 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wtt8f" podUID="da7e45bf-dfea-414e-8f0b-98242e6d10f5" Jul 30 08:12:19.287306 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.286782 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:19.287306 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.287121 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkhqs" podUID="f661b99d-6531-403f-8622-dec42ada1174" Jul 30 08:12:19.287306 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.287185 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp7kt" podUID="b8c84511-af0e-41e8-bbaf-0cb318a69dce" Jul 30 08:12:19.484574 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.484544 2576 generic.go:356] "Generic (PLEG): container finished" podID="67e8b7f3-fb6e-4551-ae93-189127b4eea7" containerID="60876ac26008a27e2450e6fad835e1c061a7899eca8d9fb5f4b96ddd109f31c6" exitCode=0 Jul 30 08:12:19.484687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.484594 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpgxk" event={"ID":"67e8b7f3-fb6e-4551-ae93-189127b4eea7","Type":"ContainerDied","Data":"60876ac26008a27e2450e6fad835e1c061a7899eca8d9fb5f4b96ddd109f31c6"} Jul 30 08:12:19.525508 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.525482 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-119.ec2.internal" event="NodeReady" Jul 30 08:12:19.525658 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.525610 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 30 08:12:19.581141 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.581108 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qfcjq"] Jul 30 08:12:19.583130 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.583108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.587964 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.587938 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kdwtn"] Jul 30 08:12:19.588180 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.588162 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 30 08:12:19.588245 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.588165 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 30 08:12:19.589774 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.589706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kdwtn" Jul 30 08:12:19.590203 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.590185 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rrdcg\"" Jul 30 08:12:19.592632 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.592612 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 30 08:12:19.592706 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.592619 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 30 08:12:19.592934 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.592918 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 30 08:12:19.592993 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.592920 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pchcs\"" Jul 30 08:12:19.601857 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.601837 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qfcjq"] Jul 30 08:12:19.617182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.616454 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kdwtn"] Jul 30 08:12:19.683207 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.683174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svkxw\" (UniqueName: \"kubernetes.io/projected/4fa47875-28d6-4eb9-86a1-1f9747488d9c-kube-api-access-svkxw\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.683207 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.683212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfst2\" (UniqueName: \"kubernetes.io/projected/a2e7232f-ba93-4ce5-9815-9266879f62a0-kube-api-access-xfst2\") pod \"ingress-canary-kdwtn\" (UID: \"a2e7232f-ba93-4ce5-9815-9266879f62a0\") " pod="openshift-ingress-canary/ingress-canary-kdwtn" Jul 30 08:12:19.683439 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.683271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa47875-28d6-4eb9-86a1-1f9747488d9c-tmp-dir\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.683439 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.683298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa47875-28d6-4eb9-86a1-1f9747488d9c-config-volume\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.683439 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.683315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2e7232f-ba93-4ce5-9815-9266879f62a0-cert\") pod \"ingress-canary-kdwtn\" (UID: \"a2e7232f-ba93-4ce5-9815-9266879f62a0\") " pod="openshift-ingress-canary/ingress-canary-kdwtn" Jul 30 08:12:19.683439 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.683398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.784531 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.784499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa47875-28d6-4eb9-86a1-1f9747488d9c-config-volume\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.784531 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.784533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2e7232f-ba93-4ce5-9815-9266879f62a0-cert\") pod \"ingress-canary-kdwtn\" (UID: \"a2e7232f-ba93-4ce5-9815-9266879f62a0\") " pod="openshift-ingress-canary/ingress-canary-kdwtn" Jul 30 08:12:19.784719 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.784554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.784719 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.784588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svkxw\" (UniqueName: \"kubernetes.io/projected/4fa47875-28d6-4eb9-86a1-1f9747488d9c-kube-api-access-svkxw\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.784719 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.784614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfst2\" (UniqueName: \"kubernetes.io/projected/a2e7232f-ba93-4ce5-9815-9266879f62a0-kube-api-access-xfst2\") pod \"ingress-canary-kdwtn\" (UID: \"a2e7232f-ba93-4ce5-9815-9266879f62a0\") " pod="openshift-ingress-canary/ingress-canary-kdwtn" Jul 30 08:12:19.784719 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.784691 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 08:12:19.784880 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.784765 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls podName:4fa47875-28d6-4eb9-86a1-1f9747488d9c nodeName:}" failed. No retries permitted until 2026-07-30 08:12:20.284745125 +0000 UTC m=+34.544691439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls") pod "dns-default-qfcjq" (UID: "4fa47875-28d6-4eb9-86a1-1f9747488d9c") : secret "dns-default-metrics-tls" not found Jul 30 08:12:19.784880 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.784804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa47875-28d6-4eb9-86a1-1f9747488d9c-tmp-dir\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.785227 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.785207 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa47875-28d6-4eb9-86a1-1f9747488d9c-tmp-dir\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.785438 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.785421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa47875-28d6-4eb9-86a1-1f9747488d9c-config-volume\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.788956 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.788937 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2e7232f-ba93-4ce5-9815-9266879f62a0-cert\") pod \"ingress-canary-kdwtn\" (UID: \"a2e7232f-ba93-4ce5-9815-9266879f62a0\") " pod="openshift-ingress-canary/ingress-canary-kdwtn" Jul 30 08:12:19.794112 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.794090 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svkxw\" (UniqueName: \"kubernetes.io/projected/4fa47875-28d6-4eb9-86a1-1f9747488d9c-kube-api-access-svkxw\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:19.794207 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.794161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfst2\" (UniqueName: \"kubernetes.io/projected/a2e7232f-ba93-4ce5-9815-9266879f62a0-kube-api-access-xfst2\") pod \"ingress-canary-kdwtn\" (UID: \"a2e7232f-ba93-4ce5-9815-9266879f62a0\") " pod="openshift-ingress-canary/ingress-canary-kdwtn" Jul 30 08:12:19.898228 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.898136 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kdwtn" Jul 30 08:12:19.987017 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.986992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:19.987131 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.987029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:19.987131 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:19.987063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh\") pod \"network-check-target-xp7kt\" (UID: \"b8c84511-af0e-41e8-bbaf-0cb318a69dce\") " pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:19.987258 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.987247 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 08:12:19.987314 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.987265 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 08:12:19.987314 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.987274 2576 projected.go:196] Error preparing data for projected volume kube-api-access-sm6dh for pod openshift-network-diagnostics/network-check-target-xp7kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:12:19.987314 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.987305 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 08:12:19.987314 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.987321 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh podName:b8c84511-af0e-41e8-bbaf-0cb318a69dce nodeName:}" failed. No retries permitted until 2026-07-30 08:12:51.987308763 +0000 UTC m=+66.247255077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sm6dh" (UniqueName: "kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh") pod "network-check-target-xp7kt" (UID: "b8c84511-af0e-41e8-bbaf-0cb318a69dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 08:12:19.987524 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.987242 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:12:19.987524 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.987426 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret podName:da7e45bf-dfea-414e-8f0b-98242e6d10f5 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:51.987363914 +0000 UTC m=+66.247310242 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret") pod "global-pull-secret-syncer-wtt8f" (UID: "da7e45bf-dfea-414e-8f0b-98242e6d10f5") : object "kube-system"/"original-pull-secret" not registered Jul 30 08:12:19.987524 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:19.987449 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs podName:f661b99d-6531-403f-8622-dec42ada1174 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:51.987437838 +0000 UTC m=+66.247384159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs") pod "network-metrics-daemon-rkhqs" (UID: "f661b99d-6531-403f-8622-dec42ada1174") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 08:12:20.031568 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:20.031537 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kdwtn"] Jul 30 08:12:20.034802 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:12:20.034772 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e7232f_ba93_4ce5_9815_9266879f62a0.slice/crio-b59afe3120a985950b9a80a4eebb20d0544439fb0a983dbc6cd4c9f50756a387 WatchSource:0}: Error finding container b59afe3120a985950b9a80a4eebb20d0544439fb0a983dbc6cd4c9f50756a387: Status 404 returned error can't find the container with id b59afe3120a985950b9a80a4eebb20d0544439fb0a983dbc6cd4c9f50756a387 Jul 30 08:12:20.288612 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:20.288588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:20.289288 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:20.288703 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 08:12:20.289288 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:20.288780 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls podName:4fa47875-28d6-4eb9-86a1-1f9747488d9c nodeName:}" failed. No retries permitted until 2026-07-30 08:12:21.288761146 +0000 UTC m=+35.548707480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls") pod "dns-default-qfcjq" (UID: "4fa47875-28d6-4eb9-86a1-1f9747488d9c") : secret "dns-default-metrics-tls" not found Jul 30 08:12:20.488068 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:20.488027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kdwtn" event={"ID":"a2e7232f-ba93-4ce5-9815-9266879f62a0","Type":"ContainerStarted","Data":"b59afe3120a985950b9a80a4eebb20d0544439fb0a983dbc6cd4c9f50756a387"} Jul 30 08:12:21.286099 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:21.286049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:21.286294 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:21.286164 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:21.286294 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:21.286168 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:21.288944 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:21.288929 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 08:12:21.289260 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:21.288981 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 08:12:21.289963 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:21.289950 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t6dmg\"" Jul 30 08:12:21.290165 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:21.290149 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 08:12:21.290200 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:21.290168 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6z2n4\"" Jul 30 08:12:21.290247 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:21.290210 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 08:12:21.297976 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:21.297951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:21.298095 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:21.298079 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 08:12:21.298174 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:21.298163 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls podName:4fa47875-28d6-4eb9-86a1-1f9747488d9c nodeName:}" failed. No retries permitted until 2026-07-30 08:12:23.298142949 +0000 UTC m=+37.558089281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls") pod "dns-default-qfcjq" (UID: "4fa47875-28d6-4eb9-86a1-1f9747488d9c") : secret "dns-default-metrics-tls" not found Jul 30 08:12:23.316340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:23.316268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:23.316875 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:23.316472 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 08:12:23.316875 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:23.316547 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls podName:4fa47875-28d6-4eb9-86a1-1f9747488d9c nodeName:}" failed. No retries permitted until 2026-07-30 08:12:27.316527111 +0000 UTC m=+41.576473445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls") pod "dns-default-qfcjq" (UID: "4fa47875-28d6-4eb9-86a1-1f9747488d9c") : secret "dns-default-metrics-tls" not found Jul 30 08:12:26.506462 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:26.506225 2576 generic.go:356] "Generic (PLEG): container finished" podID="67e8b7f3-fb6e-4551-ae93-189127b4eea7" containerID="505c86c6b2166776d04a3342bad7e9bc57f2ae5649da74e2c89ebba1324574f1" exitCode=0 Jul 30 08:12:26.506895 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:26.506309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpgxk" event={"ID":"67e8b7f3-fb6e-4551-ae93-189127b4eea7","Type":"ContainerDied","Data":"505c86c6b2166776d04a3342bad7e9bc57f2ae5649da74e2c89ebba1324574f1"} Jul 30 08:12:26.507973 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:26.507949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kdwtn" event={"ID":"a2e7232f-ba93-4ce5-9815-9266879f62a0","Type":"ContainerStarted","Data":"3af1591d5fe01a69e17120170db6864ea071274e829c4a046ec79b73aaac1f2b"} Jul 30 08:12:26.548008 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:26.547961 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kdwtn" podStartSLOduration=1.983022114 podStartE2EDuration="7.547944569s" totalImagesPullingTime="5.564922455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:26.547360098 +0000 UTC m=+40.807306455" watchObservedRunningTime="2026-07-30 08:12:26.547944569 +0000 UTC m=+40.807890905" Jul 30 08:12:27.345259 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:27.345217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:27.345467 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:27.345358 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 08:12:27.345467 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:27.345444 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls podName:4fa47875-28d6-4eb9-86a1-1f9747488d9c nodeName:}" failed. No retries permitted until 2026-07-30 08:12:35.345428382 +0000 UTC m=+49.605374695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls") pod "dns-default-qfcjq" (UID: "4fa47875-28d6-4eb9-86a1-1f9747488d9c") : secret "dns-default-metrics-tls" not found Jul 30 08:12:27.513313 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:27.513280 2576 generic.go:356] "Generic (PLEG): container finished" podID="67e8b7f3-fb6e-4551-ae93-189127b4eea7" containerID="b2cd64a313e734cc79b98af726c9a0446a3eb422925a33b901a975d4cf732e34" exitCode=0 Jul 30 08:12:27.513689 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:27.513385 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpgxk" event={"ID":"67e8b7f3-fb6e-4551-ae93-189127b4eea7","Type":"ContainerDied","Data":"b2cd64a313e734cc79b98af726c9a0446a3eb422925a33b901a975d4cf732e34"} Jul 30 08:12:28.518315 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:28.518285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpgxk" event={"ID":"67e8b7f3-fb6e-4551-ae93-189127b4eea7","Type":"ContainerStarted","Data":"409dfcaedd0baf5dec333869090fbdbfee3f3c6b5a0600549b9d8505fa57dba6"} Jul 30 08:12:28.543400 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:28.543344 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zpgxk" podStartSLOduration=8.073679992 podStartE2EDuration="42.543330435s" totalImagesPullingTime="34.469650443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:28.542081895 +0000 UTC m=+42.802028231" watchObservedRunningTime="2026-07-30 08:12:28.543330435 +0000 UTC m=+42.803276771" Jul 30 08:12:35.405599 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:35.405557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:35.406086 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:35.405752 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 08:12:35.406086 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:35.405842 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls podName:4fa47875-28d6-4eb9-86a1-1f9747488d9c nodeName:}" failed. No retries permitted until 2026-07-30 08:12:51.405819303 +0000 UTC m=+65.665765634 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls") pod "dns-default-qfcjq" (UID: "4fa47875-28d6-4eb9-86a1-1f9747488d9c") : secret "dns-default-metrics-tls" not found Jul 30 08:12:42.605750 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.605715 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-75lsm"] Jul 30 08:12:42.607494 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.607476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.613986 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.613963 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 30 08:12:42.614104 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.614065 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 30 08:12:42.614104 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.614068 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 30 08:12:42.615309 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.615287 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-65lbm\"" Jul 30 08:12:42.615408 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.615287 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 30 08:12:42.622208 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.622179 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 30 08:12:42.622537 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.622521 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-68585dd8ff-bhdlw"] Jul 30 08:12:42.624494 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.624472 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-75lsm"] Jul 30 08:12:42.624583 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.624551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.627220 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.627116 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 30 08:12:42.627220 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.627125 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 30 08:12:42.627383 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.627282 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 30 08:12:42.627383 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.627291 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 30 08:12:42.627493 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.627393 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 30 08:12:42.627493 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.627439 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8w4sn\"" Jul 30 08:12:42.627648 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.627636 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 30 08:12:42.638556 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.638534 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68585dd8ff-bhdlw"] Jul 30 08:12:42.657157 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.657128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0ed27260-2f69-4fba-8783-e7155daa29e5-snapshots\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.657279 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.657166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-default-certificate\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.657279 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.657185 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-stats-auth\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.657279 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.657205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ed27260-2f69-4fba-8783-e7155daa29e5-service-ca-bundle\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.657279 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.657219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ed27260-2f69-4fba-8783-e7155daa29e5-serving-cert\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.657279 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.657270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ed27260-2f69-4fba-8783-e7155daa29e5-tmp\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.657482 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.657341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wddc6\" (UniqueName: \"kubernetes.io/projected/b835a639-ac08-499a-9b99-79744d0e8e1b-kube-api-access-wddc6\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.657482 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.657367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnd8f\" (UniqueName: \"kubernetes.io/projected/0ed27260-2f69-4fba-8783-e7155daa29e5-kube-api-access-rnd8f\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.657482 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.657431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.657482 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.657455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ed27260-2f69-4fba-8783-e7155daa29e5-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.657482 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.657470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.758798 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.758761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wddc6\" (UniqueName: \"kubernetes.io/projected/b835a639-ac08-499a-9b99-79744d0e8e1b-kube-api-access-wddc6\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.758798 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.758797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnd8f\" (UniqueName: \"kubernetes.io/projected/0ed27260-2f69-4fba-8783-e7155daa29e5-kube-api-access-rnd8f\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.759017 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.758943 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.759017 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.758981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ed27260-2f69-4fba-8783-e7155daa29e5-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.759017 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.759008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.759122 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:42.759086 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:12:43.259064285 +0000 UTC m=+57.519010608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : configmap references non-existent config key: service-ca.crt Jul 30 08:12:42.759171 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.759126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0ed27260-2f69-4fba-8783-e7155daa29e5-snapshots\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.759171 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:42.759147 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 08:12:42.759257 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.759189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-default-certificate\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.759257 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:42.759218 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:12:43.259198917 +0000 UTC m=+57.519145247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : secret "router-metrics-certs-default" not found Jul 30 08:12:42.759358 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.759253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-stats-auth\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.759358 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.759291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ed27260-2f69-4fba-8783-e7155daa29e5-service-ca-bundle\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.759358 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.759328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ed27260-2f69-4fba-8783-e7155daa29e5-serving-cert\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.759519 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.759361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ed27260-2f69-4fba-8783-e7155daa29e5-tmp\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.759894 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.759868 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ed27260-2f69-4fba-8783-e7155daa29e5-tmp\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.759980 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.759891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0ed27260-2f69-4fba-8783-e7155daa29e5-snapshots\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.760017 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.759989 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ed27260-2f69-4fba-8783-e7155daa29e5-service-ca-bundle\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.760615 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.760591 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ed27260-2f69-4fba-8783-e7155daa29e5-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.762331 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.762309 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ed27260-2f69-4fba-8783-e7155daa29e5-serving-cert\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.762428 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.762401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-stats-auth\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.762506 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.762487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-default-certificate\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.767445 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.767420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wddc6\" (UniqueName: \"kubernetes.io/projected/b835a639-ac08-499a-9b99-79744d0e8e1b-kube-api-access-wddc6\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:42.767732 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.767711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnd8f\" (UniqueName: \"kubernetes.io/projected/0ed27260-2f69-4fba-8783-e7155daa29e5-kube-api-access-rnd8f\") pod \"insights-operator-5885fdb759-75lsm\" (UID: \"0ed27260-2f69-4fba-8783-e7155daa29e5\") " pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:42.916234 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:42.916145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-75lsm" Jul 30 08:12:43.037556 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:43.037522 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-75lsm"] Jul 30 08:12:43.040486 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:12:43.040460 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed27260_2f69_4fba_8783_e7155daa29e5.slice/crio-996b0cae3890fe2bef34f0c23a6888b09bb43452f854c1dc3c9345ef5ce4d661 WatchSource:0}: Error finding container 996b0cae3890fe2bef34f0c23a6888b09bb43452f854c1dc3c9345ef5ce4d661: Status 404 returned error can't find the container with id 996b0cae3890fe2bef34f0c23a6888b09bb43452f854c1dc3c9345ef5ce4d661 Jul 30 08:12:43.264363 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:43.264266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:43.264363 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:43.264306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:43.264611 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:43.264435 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 08:12:43.264611 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:43.264491 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:12:44.264467284 +0000 UTC m=+58.524413615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : configmap references non-existent config key: service-ca.crt Jul 30 08:12:43.264611 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:43.264519 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:12:44.264510621 +0000 UTC m=+58.524456935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : secret "router-metrics-certs-default" not found Jul 30 08:12:43.554429 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:43.554339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-75lsm" event={"ID":"0ed27260-2f69-4fba-8783-e7155daa29e5","Type":"ContainerStarted","Data":"996b0cae3890fe2bef34f0c23a6888b09bb43452f854c1dc3c9345ef5ce4d661"} Jul 30 08:12:44.275024 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.274982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:44.275024 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.275031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:44.275519 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:44.275161 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:12:46.275142849 +0000 UTC m=+60.535089167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : configmap references non-existent config key: service-ca.crt Jul 30 08:12:44.275519 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:44.275211 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 08:12:44.275519 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:44.275267 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:12:46.275251407 +0000 UTC m=+60.535197721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : secret "router-metrics-certs-default" not found Jul 30 08:12:44.545320 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.545282 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-c9mnf"] Jul 30 08:12:44.547564 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.547538 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.550117 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.550094 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-r887q\"" Jul 30 08:12:44.550339 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.550308 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 30 08:12:44.551389 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.551199 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 30 08:12:44.551389 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.551215 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 30 08:12:44.551389 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.551215 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 30 08:12:44.560531 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.558471 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-c9mnf"] Jul 30 08:12:44.560531 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.559954 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 30 08:12:44.577908 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.577878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pllng\" (UniqueName: \"kubernetes.io/projected/9e1bf430-3c04-4542-87b6-f0f29509e277-kube-api-access-pllng\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.578047 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.577920 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e1bf430-3c04-4542-87b6-f0f29509e277-config\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.578047 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.577940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e1bf430-3c04-4542-87b6-f0f29509e277-trusted-ca\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.578047 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.577968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e1bf430-3c04-4542-87b6-f0f29509e277-serving-cert\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.679360 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.679321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pllng\" (UniqueName: \"kubernetes.io/projected/9e1bf430-3c04-4542-87b6-f0f29509e277-kube-api-access-pllng\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.679562 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.679417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e1bf430-3c04-4542-87b6-f0f29509e277-config\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.679562 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.679446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e1bf430-3c04-4542-87b6-f0f29509e277-trusted-ca\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.679562 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.679478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e1bf430-3c04-4542-87b6-f0f29509e277-serving-cert\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.680311 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.680279 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e1bf430-3c04-4542-87b6-f0f29509e277-config\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.680548 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.680525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e1bf430-3c04-4542-87b6-f0f29509e277-trusted-ca\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.682158 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.682137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e1bf430-3c04-4542-87b6-f0f29509e277-serving-cert\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.688623 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.688595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pllng\" (UniqueName: \"kubernetes.io/projected/9e1bf430-3c04-4542-87b6-f0f29509e277-kube-api-access-pllng\") pod \"console-operator-584dcb6b54-c9mnf\" (UID: \"9e1bf430-3c04-4542-87b6-f0f29509e277\") " pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:44.863451 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:44.863359 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:45.485476 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:45.485432 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzfgq" Jul 30 08:12:45.519739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:45.519713 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-c9mnf"] Jul 30 08:12:45.522910 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:12:45.522874 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1bf430_3c04_4542_87b6_f0f29509e277.slice/crio-452842c3c1aa427e0e941d0a8abf3f9889724c8633107e301564e68c3a21f021 WatchSource:0}: Error finding container 452842c3c1aa427e0e941d0a8abf3f9889724c8633107e301564e68c3a21f021: Status 404 returned error can't find the container with id 452842c3c1aa427e0e941d0a8abf3f9889724c8633107e301564e68c3a21f021 Jul 30 08:12:45.562449 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:45.562417 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" event={"ID":"9e1bf430-3c04-4542-87b6-f0f29509e277","Type":"ContainerStarted","Data":"452842c3c1aa427e0e941d0a8abf3f9889724c8633107e301564e68c3a21f021"} Jul 30 08:12:46.292958 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.292739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:46.292958 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.292783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:46.292958 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:46.292877 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:12:50.292858976 +0000 UTC m=+64.552805317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : configmap references non-existent config key: service-ca.crt Jul 30 08:12:46.292958 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:46.292901 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 08:12:46.292958 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:46.292935 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:12:50.292924576 +0000 UTC m=+64.552870895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : secret "router-metrics-certs-default" not found Jul 30 08:12:46.566631 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.566540 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-75lsm" event={"ID":"0ed27260-2f69-4fba-8783-e7155daa29e5","Type":"ContainerStarted","Data":"aeacc1b86ab47c148cf5c8b22fa95c1bed491e8b413e9a10ba71abc98df3044b"} Jul 30 08:12:46.592064 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.591994 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-75lsm" podStartSLOduration=1.952526721 podStartE2EDuration="4.591973163s" totalImagesPullingTime="2.639446442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:46.591596376 +0000 UTC m=+60.851542712" watchObservedRunningTime="2026-07-30 08:12:46.591973163 +0000 UTC m=+60.851919500" Jul 30 08:12:46.695847 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.695819 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4twmv"] Jul 30 08:12:46.698565 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.698534 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.702309 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.702288 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 30 08:12:46.702955 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.702934 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 30 08:12:46.703107 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.703081 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zqlsf\"" Jul 30 08:12:46.715511 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.715491 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4twmv"] Jul 30 08:12:46.799281 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.799243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.799281 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.799282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.799515 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.799336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7069b820-c097-4267-94ed-7c7d5ca4f8d3-data-volume\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.799515 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.799357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xj7\" (UniqueName: \"kubernetes.io/projected/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-api-access-p4xj7\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.799515 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.799488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7069b820-c097-4267-94ed-7c7d5ca4f8d3-crio-socket\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.900501 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.900407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7069b820-c097-4267-94ed-7c7d5ca4f8d3-crio-socket\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.900655 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.900547 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7069b820-c097-4267-94ed-7c7d5ca4f8d3-crio-socket\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.900655 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.900576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.900655 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.900609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.900793 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.900665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7069b820-c097-4267-94ed-7c7d5ca4f8d3-data-volume\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.900793 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.900688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xj7\" (UniqueName: \"kubernetes.io/projected/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-api-access-p4xj7\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.900793 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:46.900720 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 08:12:46.900941 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:46.900799 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm podName:7069b820-c097-4267-94ed-7c7d5ca4f8d3 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:47.400776492 +0000 UTC m=+61.660722810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4twmv" (UID: "7069b820-c097-4267-94ed-7c7d5ca4f8d3") : configmap "kube-rbac-proxy" not found Jul 30 08:12:46.900941 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:46.900807 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 08:12:46.900941 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:46.900860 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls podName:7069b820-c097-4267-94ed-7c7d5ca4f8d3 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:47.400844058 +0000 UTC m=+61.660790386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4twmv" (UID: "7069b820-c097-4267-94ed-7c7d5ca4f8d3") : secret "insights-runtime-extractor-tls" not found Jul 30 08:12:46.901183 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.901164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7069b820-c097-4267-94ed-7c7d5ca4f8d3-data-volume\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:46.909591 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:46.909566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xj7\" (UniqueName: \"kubernetes.io/projected/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-api-access-p4xj7\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:47.404639 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:47.404603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:47.404639 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:47.404650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:47.404852 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:47.404742 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 08:12:47.404852 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:47.404814 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm podName:7069b820-c097-4267-94ed-7c7d5ca4f8d3 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:48.40479458 +0000 UTC m=+62.664740908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4twmv" (UID: "7069b820-c097-4267-94ed-7c7d5ca4f8d3") : configmap "kube-rbac-proxy" not found Jul 30 08:12:47.404852 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:47.404824 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 08:12:47.405010 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:47.404883 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls podName:7069b820-c097-4267-94ed-7c7d5ca4f8d3 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:48.404865647 +0000 UTC m=+62.664811977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4twmv" (UID: "7069b820-c097-4267-94ed-7c7d5ca4f8d3") : secret "insights-runtime-extractor-tls" not found Jul 30 08:12:48.411936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:48.411891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:48.411936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:48.411939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:48.412465 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:48.412021 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 08:12:48.412465 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:48.412089 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm podName:7069b820-c097-4267-94ed-7c7d5ca4f8d3 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:50.412073039 +0000 UTC m=+64.672019357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4twmv" (UID: "7069b820-c097-4267-94ed-7c7d5ca4f8d3") : configmap "kube-rbac-proxy" not found Jul 30 08:12:48.412465 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:48.412119 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 08:12:48.412465 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:48.412172 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls podName:7069b820-c097-4267-94ed-7c7d5ca4f8d3 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:50.412159327 +0000 UTC m=+64.672105641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4twmv" (UID: "7069b820-c097-4267-94ed-7c7d5ca4f8d3") : secret "insights-runtime-extractor-tls" not found Jul 30 08:12:48.573744 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:48.573713 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/0.log" Jul 30 08:12:48.573933 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:48.573762 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e1bf430-3c04-4542-87b6-f0f29509e277" containerID="af30fb330ae0368a1ded0dce2534073eb5d4b49f5a1ada662e6663a2e6cff40f" exitCode=255 Jul 30 08:12:48.573933 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:48.573886 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" event={"ID":"9e1bf430-3c04-4542-87b6-f0f29509e277","Type":"ContainerDied","Data":"af30fb330ae0368a1ded0dce2534073eb5d4b49f5a1ada662e6663a2e6cff40f"} Jul 30 08:12:48.574150 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:48.574135 2576 scope.go:117] "RemoveContainer" containerID="af30fb330ae0368a1ded0dce2534073eb5d4b49f5a1ada662e6663a2e6cff40f" Jul 30 08:12:48.680238 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:48.680217 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lp4ct_bd9d1bb7-c615-40e0-b53a-d2ca26386d62/dns-node-resolver/0.log" Jul 30 08:12:49.577700 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:49.577671 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:12:49.578083 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:49.578051 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/0.log" Jul 30 08:12:49.578147 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:49.578084 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e1bf430-3c04-4542-87b6-f0f29509e277" containerID="93e85a1b677a070918c9e4d3734e910ea4e79def85ede1806c4c1274cd098b4e" exitCode=255 Jul 30 08:12:49.578147 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:49.578114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" event={"ID":"9e1bf430-3c04-4542-87b6-f0f29509e277","Type":"ContainerDied","Data":"93e85a1b677a070918c9e4d3734e910ea4e79def85ede1806c4c1274cd098b4e"} Jul 30 08:12:49.578228 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:49.578186 2576 scope.go:117] "RemoveContainer" containerID="af30fb330ae0368a1ded0dce2534073eb5d4b49f5a1ada662e6663a2e6cff40f" Jul 30 08:12:49.578358 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:49.578341 2576 scope.go:117] "RemoveContainer" containerID="93e85a1b677a070918c9e4d3734e910ea4e79def85ede1806c4c1274cd098b4e" Jul 30 08:12:49.578557 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:49.578534 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-c9mnf_openshift-console-operator(9e1bf430-3c04-4542-87b6-f0f29509e277)\"" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" podUID="9e1bf430-3c04-4542-87b6-f0f29509e277" Jul 30 08:12:49.679870 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:49.679842 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s6r42_ab4c1b36-d20a-4ac3-96f3-fcd94d257061/node-ca/0.log" Jul 30 08:12:50.328734 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:50.328702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:50.328734 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:50.328735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:50.328956 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:50.328837 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 08:12:50.328956 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:50.328863 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:12:58.328845601 +0000 UTC m=+72.588791931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : configmap references non-existent config key: service-ca.crt Jul 30 08:12:50.328956 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:50.328923 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:12:58.328916977 +0000 UTC m=+72.588863291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : secret "router-metrics-certs-default" not found Jul 30 08:12:50.429679 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:50.429642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:50.429845 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:50.429686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:50.429845 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:50.429763 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 08:12:50.429845 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:50.429815 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 08:12:50.429845 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:50.429828 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm podName:7069b820-c097-4267-94ed-7c7d5ca4f8d3 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:54.42981453 +0000 UTC m=+68.689760881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4twmv" (UID: "7069b820-c097-4267-94ed-7c7d5ca4f8d3") : configmap "kube-rbac-proxy" not found Jul 30 08:12:50.430001 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:50.429864 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls podName:7069b820-c097-4267-94ed-7c7d5ca4f8d3 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:54.429851944 +0000 UTC m=+68.689798258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4twmv" (UID: "7069b820-c097-4267-94ed-7c7d5ca4f8d3") : secret "insights-runtime-extractor-tls" not found Jul 30 08:12:50.480357 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:50.480326 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kdwtn_a2e7232f-ba93-4ce5-9815-9266879f62a0/serve-healthcheck-canary/0.log" Jul 30 08:12:50.582319 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:50.582237 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:12:50.582697 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:50.582569 2576 scope.go:117] "RemoveContainer" containerID="93e85a1b677a070918c9e4d3734e910ea4e79def85ede1806c4c1274cd098b4e" Jul 30 08:12:50.582736 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:50.582720 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-c9mnf_openshift-console-operator(9e1bf430-3c04-4542-87b6-f0f29509e277)\"" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" podUID="9e1bf430-3c04-4542-87b6-f0f29509e277" Jul 30 08:12:51.439664 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:51.439621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:12:51.439848 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:51.439777 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 08:12:51.439848 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:51.439843 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls podName:4fa47875-28d6-4eb9-86a1-1f9747488d9c nodeName:}" failed. No retries permitted until 2026-07-30 08:13:23.439825836 +0000 UTC m=+97.699772150 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls") pod "dns-default-qfcjq" (UID: "4fa47875-28d6-4eb9-86a1-1f9747488d9c") : secret "dns-default-metrics-tls" not found Jul 30 08:12:52.044158 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.044125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh\") pod \"network-check-target-xp7kt\" (UID: \"b8c84511-af0e-41e8-bbaf-0cb318a69dce\") " pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:52.044568 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.044216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:12:52.044568 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.044264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:52.046939 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.046921 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 08:12:52.046989 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.046953 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 08:12:52.047075 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.047059 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 08:12:52.055293 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:52.055272 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 30 08:12:52.055394 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:52.055325 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs podName:f661b99d-6531-403f-8622-dec42ada1174 nodeName:}" failed. No retries permitted until 2026-07-30 08:13:56.055310158 +0000 UTC m=+130.315256472 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs") pod "network-metrics-daemon-rkhqs" (UID: "f661b99d-6531-403f-8622-dec42ada1174") : secret "metrics-daemon-secret" not found Jul 30 08:12:52.057533 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.057502 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7e45bf-dfea-414e-8f0b-98242e6d10f5-original-pull-secret\") pod \"global-pull-secret-syncer-wtt8f\" (UID: \"da7e45bf-dfea-414e-8f0b-98242e6d10f5\") " pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:52.057633 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.057603 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 08:12:52.068320 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.068293 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/b8c84511-af0e-41e8-bbaf-0cb318a69dce-kube-api-access-sm6dh\") pod \"network-check-target-xp7kt\" (UID: \"b8c84511-af0e-41e8-bbaf-0cb318a69dce\") " pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:52.199568 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.199542 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6z2n4\"" Jul 30 08:12:52.202642 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.202623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wtt8f" Jul 30 08:12:52.207467 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.207449 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:52.336429 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.336358 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wtt8f"] Jul 30 08:12:52.339337 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:12:52.339299 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7e45bf_dfea_414e_8f0b_98242e6d10f5.slice/crio-1e102eda04a387d9825834ac72dc6d9b97e9f1af27a2318ffbc6ec82bb82647d WatchSource:0}: Error finding container 1e102eda04a387d9825834ac72dc6d9b97e9f1af27a2318ffbc6ec82bb82647d: Status 404 returned error can't find the container with id 1e102eda04a387d9825834ac72dc6d9b97e9f1af27a2318ffbc6ec82bb82647d Jul 30 08:12:52.351357 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.351335 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xp7kt"] Jul 30 08:12:52.354118 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:12:52.354096 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c84511_af0e_41e8_bbaf_0cb318a69dce.slice/crio-00ec92938d603301f0d1080755de2b5fabedc20c94e54c7efbefd7b3a799ef84 WatchSource:0}: Error finding container 00ec92938d603301f0d1080755de2b5fabedc20c94e54c7efbefd7b3a799ef84: Status 404 returned error can't find the container with id 00ec92938d603301f0d1080755de2b5fabedc20c94e54c7efbefd7b3a799ef84 Jul 30 08:12:52.588636 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.588552 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xp7kt" event={"ID":"b8c84511-af0e-41e8-bbaf-0cb318a69dce","Type":"ContainerStarted","Data":"00ec92938d603301f0d1080755de2b5fabedc20c94e54c7efbefd7b3a799ef84"} Jul 30 08:12:52.589605 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.589577 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wtt8f" event={"ID":"da7e45bf-dfea-414e-8f0b-98242e6d10f5","Type":"ContainerStarted","Data":"1e102eda04a387d9825834ac72dc6d9b97e9f1af27a2318ffbc6ec82bb82647d"} Jul 30 08:12:52.775544 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.775512 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7"] Jul 30 08:12:52.777594 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.777567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7" Jul 30 08:12:52.780211 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.780044 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 30 08:12:52.780211 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.780044 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 30 08:12:52.780211 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.780108 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hd8c2\"" Jul 30 08:12:52.790827 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.790793 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7"] Jul 30 08:12:52.851946 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.851851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7jx\" (UniqueName: \"kubernetes.io/projected/38c45b61-22fb-4184-8653-f624b4fe9761-kube-api-access-jg7jx\") pod \"migrator-6865c9967-8qtn7\" (UID: \"38c45b61-22fb-4184-8653-f624b4fe9761\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7" Jul 30 08:12:52.953170 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.953124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7jx\" (UniqueName: \"kubernetes.io/projected/38c45b61-22fb-4184-8653-f624b4fe9761-kube-api-access-jg7jx\") pod \"migrator-6865c9967-8qtn7\" (UID: \"38c45b61-22fb-4184-8653-f624b4fe9761\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7" Jul 30 08:12:52.961735 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:52.961703 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7jx\" (UniqueName: \"kubernetes.io/projected/38c45b61-22fb-4184-8653-f624b4fe9761-kube-api-access-jg7jx\") pod \"migrator-6865c9967-8qtn7\" (UID: \"38c45b61-22fb-4184-8653-f624b4fe9761\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7" Jul 30 08:12:53.088193 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:53.088157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7" Jul 30 08:12:53.243726 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:53.243693 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7"] Jul 30 08:12:53.247344 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:12:53.247314 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c45b61_22fb_4184_8653_f624b4fe9761.slice/crio-5653829a6974b26afd3e6ef03add7837b2b6af4911df41c033322ac51e33ea1b WatchSource:0}: Error finding container 5653829a6974b26afd3e6ef03add7837b2b6af4911df41c033322ac51e33ea1b: Status 404 returned error can't find the container with id 5653829a6974b26afd3e6ef03add7837b2b6af4911df41c033322ac51e33ea1b Jul 30 08:12:53.594794 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:53.594697 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7" event={"ID":"38c45b61-22fb-4184-8653-f624b4fe9761","Type":"ContainerStarted","Data":"5653829a6974b26afd3e6ef03add7837b2b6af4911df41c033322ac51e33ea1b"} Jul 30 08:12:54.468319 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:54.468109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:54.468319 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:54.468222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:12:54.468319 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:54.468322 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 08:12:54.468905 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:54.468406 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm podName:7069b820-c097-4267-94ed-7c7d5ca4f8d3 nodeName:}" failed. No retries permitted until 2026-07-30 08:13:02.468368563 +0000 UTC m=+76.728314877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4twmv" (UID: "7069b820-c097-4267-94ed-7c7d5ca4f8d3") : configmap "kube-rbac-proxy" not found Jul 30 08:12:54.468905 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:54.468334 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 08:12:54.468905 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:54.468544 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls podName:7069b820-c097-4267-94ed-7c7d5ca4f8d3 nodeName:}" failed. No retries permitted until 2026-07-30 08:13:02.468522061 +0000 UTC m=+76.728468385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4twmv" (UID: "7069b820-c097-4267-94ed-7c7d5ca4f8d3") : secret "insights-runtime-extractor-tls" not found Jul 30 08:12:54.864494 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:54.864454 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:54.864670 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:54.864516 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:12:54.864935 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:54.864915 2576 scope.go:117] "RemoveContainer" containerID="93e85a1b677a070918c9e4d3734e910ea4e79def85ede1806c4c1274cd098b4e" Jul 30 08:12:54.865097 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:54.865076 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-c9mnf_openshift-console-operator(9e1bf430-3c04-4542-87b6-f0f29509e277)\"" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" podUID="9e1bf430-3c04-4542-87b6-f0f29509e277" Jul 30 08:12:55.069242 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.069208 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b8bbf9d84-drkkr"] Jul 30 08:12:55.071422 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.071400 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.073741 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.073709 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 30 08:12:55.074093 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.073946 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 30 08:12:55.074093 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.073976 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-phtjq\"" Jul 30 08:12:55.074093 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.074024 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 30 08:12:55.080513 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.080493 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 30 08:12:55.083762 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.083732 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b8bbf9d84-drkkr"] Jul 30 08:12:55.175131 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.175043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-certificates\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.175288 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.175184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-bound-sa-token\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.175288 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.175211 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-trusted-ca\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.175288 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.175236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-image-registry-private-configuration\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.175508 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.175336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8dccc1d2-4495-4284-8f11-bf0e283dabb9-ca-trust-extracted\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.175508 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.175392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8gg\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-kube-api-access-cs8gg\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.175508 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.175429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-installation-pull-secrets\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.175508 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.175462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.276911 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.276875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-bound-sa-token\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.277146 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.276918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-trusted-ca\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.277146 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.276945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-image-registry-private-configuration\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.277252 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.277140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8dccc1d2-4495-4284-8f11-bf0e283dabb9-ca-trust-extracted\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.277252 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.277189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8gg\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-kube-api-access-cs8gg\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.277252 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.277220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-installation-pull-secrets\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.277252 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.277247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.277475 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.277296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-certificates\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.277704 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.277673 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8dccc1d2-4495-4284-8f11-bf0e283dabb9-ca-trust-extracted\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.277803 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:55.277789 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 08:12:55.277855 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:55.277805 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8bbf9d84-drkkr: secret "image-registry-tls" not found Jul 30 08:12:55.277904 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:55.277864 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls podName:8dccc1d2-4495-4284-8f11-bf0e283dabb9 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:55.777845184 +0000 UTC m=+70.037791518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls") pod "image-registry-7b8bbf9d84-drkkr" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9") : secret "image-registry-tls" not found Jul 30 08:12:55.277985 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.277963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-certificates\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.278049 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.277999 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-trusted-ca\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.280109 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.280084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-installation-pull-secrets\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.280219 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.280140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-image-registry-private-configuration\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.286282 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.286258 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-bound-sa-token\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.286451 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.286432 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8gg\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-kube-api-access-cs8gg\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.782688 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:55.782645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:55.783114 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:55.782806 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 08:12:55.783114 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:55.782824 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8bbf9d84-drkkr: secret "image-registry-tls" not found Jul 30 08:12:55.783114 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:55.782916 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls podName:8dccc1d2-4495-4284-8f11-bf0e283dabb9 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:56.782884923 +0000 UTC m=+71.042831269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls") pod "image-registry-7b8bbf9d84-drkkr" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9") : secret "image-registry-tls" not found Jul 30 08:12:56.791850 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:56.791806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:56.792215 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:56.791970 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 08:12:56.792215 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:56.791989 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8bbf9d84-drkkr: secret "image-registry-tls" not found Jul 30 08:12:56.792215 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:56.792050 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls podName:8dccc1d2-4495-4284-8f11-bf0e283dabb9 nodeName:}" failed. No retries permitted until 2026-07-30 08:12:58.792033202 +0000 UTC m=+73.051979515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls") pod "image-registry-7b8bbf9d84-drkkr" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9") : secret "image-registry-tls" not found Jul 30 08:12:57.608154 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:57.608113 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7" event={"ID":"38c45b61-22fb-4184-8653-f624b4fe9761","Type":"ContainerStarted","Data":"ee29f34df1a4ec1ffaf338eed0c56438a36d247fae425d017c52668ca4933173"} Jul 30 08:12:57.608324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:57.608170 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7" event={"ID":"38c45b61-22fb-4184-8653-f624b4fe9761","Type":"ContainerStarted","Data":"13847e5fd68e09c75db5abd9642e6df8d990c07669b3df69fb353e63d8d4f4bf"} Jul 30 08:12:57.609564 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:57.609541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xp7kt" event={"ID":"b8c84511-af0e-41e8-bbaf-0cb318a69dce","Type":"ContainerStarted","Data":"faac48c5a9647cccad66eccc8ab1d2142bc90434fc41ad98e6e11fca21039ba7"} Jul 30 08:12:57.609670 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:57.609603 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:12:57.610942 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:57.610918 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wtt8f" event={"ID":"da7e45bf-dfea-414e-8f0b-98242e6d10f5","Type":"ContainerStarted","Data":"36c10a472a877f0a8897c9742a5c4f166ac4b2666016e55519b86e53d5091b58"} Jul 30 08:12:57.624562 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:57.624514 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8qtn7" podStartSLOduration=1.907318517 podStartE2EDuration="5.624501838s" totalImagesPullingTime="3.717183321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:57.623589207 +0000 UTC m=+71.883535544" watchObservedRunningTime="2026-07-30 08:12:57.624501838 +0000 UTC m=+71.884448171" Jul 30 08:12:57.640571 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:57.640529 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xp7kt" podStartSLOduration=67.030062402 podStartE2EDuration="1m11.640516423s" totalImagesPullingTime="4.610454021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:57.639174439 +0000 UTC m=+71.899120773" watchObservedRunningTime="2026-07-30 08:12:57.640516423 +0000 UTC m=+71.900462760" Jul 30 08:12:57.659421 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:57.659358 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wtt8f" podStartSLOduration=67.018594434 podStartE2EDuration="1m11.659343999s" totalImagesPullingTime="4.640749565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:12:57.658535736 +0000 UTC m=+71.918482071" watchObservedRunningTime="2026-07-30 08:12:57.659343999 +0000 UTC m=+71.919290351" Jul 30 08:12:58.407537 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:58.407499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:58.407921 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:58.407574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:12:58.407921 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:58.407677 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 08:12:58.407921 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:58.407716 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:13:14.407701917 +0000 UTC m=+88.667648230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : configmap references non-existent config key: service-ca.crt Jul 30 08:12:58.407921 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:58.407748 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs podName:b835a639-ac08-499a-9b99-79744d0e8e1b nodeName:}" failed. No retries permitted until 2026-07-30 08:13:14.407733962 +0000 UTC m=+88.667680278 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs") pod "router-default-68585dd8ff-bhdlw" (UID: "b835a639-ac08-499a-9b99-79744d0e8e1b") : secret "router-metrics-certs-default" not found Jul 30 08:12:58.811357 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:12:58.811315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:12:58.811587 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:58.811478 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 08:12:58.811587 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:58.811497 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8bbf9d84-drkkr: secret "image-registry-tls" not found Jul 30 08:12:58.811587 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:12:58.811554 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls podName:8dccc1d2-4495-4284-8f11-bf0e283dabb9 nodeName:}" failed. No retries permitted until 2026-07-30 08:13:02.811537519 +0000 UTC m=+77.071483856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls") pod "image-registry-7b8bbf9d84-drkkr" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9") : secret "image-registry-tls" not found Jul 30 08:13:02.542134 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:02.542091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:13:02.542573 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:02.542171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:13:02.542573 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:02.542236 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 08:13:02.542573 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:02.542310 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls podName:7069b820-c097-4267-94ed-7c7d5ca4f8d3 nodeName:}" failed. No retries permitted until 2026-07-30 08:13:18.54229324 +0000 UTC m=+92.802239556 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4twmv" (UID: "7069b820-c097-4267-94ed-7c7d5ca4f8d3") : secret "insights-runtime-extractor-tls" not found Jul 30 08:13:02.542701 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:02.542668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7069b820-c097-4267-94ed-7c7d5ca4f8d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:13:02.844353 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:02.844265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:13:02.844503 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:02.844458 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 08:13:02.844503 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:02.844473 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8bbf9d84-drkkr: secret "image-registry-tls" not found Jul 30 08:13:02.844570 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:02.844531 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls podName:8dccc1d2-4495-4284-8f11-bf0e283dabb9 nodeName:}" failed. No retries permitted until 2026-07-30 08:13:10.844514489 +0000 UTC m=+85.104460822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls") pod "image-registry-7b8bbf9d84-drkkr" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9") : secret "image-registry-tls" not found Jul 30 08:13:06.287910 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:06.287870 2576 scope.go:117] "RemoveContainer" containerID="93e85a1b677a070918c9e4d3734e910ea4e79def85ede1806c4c1274cd098b4e" Jul 30 08:13:06.639841 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:06.639819 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:13:06.640005 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:06.639871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" event={"ID":"9e1bf430-3c04-4542-87b6-f0f29509e277","Type":"ContainerStarted","Data":"eeb6e58297017e72d4ba17363acfefabb639dda42bc4d69d274853555f28a623"} Jul 30 08:13:06.640244 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:06.640221 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:13:06.658495 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:06.658453 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" podStartSLOduration=20.438116955 podStartE2EDuration="22.658440185s" totalImagesPullingTime="2.22032323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:13:06.657043039 +0000 UTC m=+80.916989375" watchObservedRunningTime="2026-07-30 08:13:06.658440185 +0000 UTC m=+80.918386519" Jul 30 08:13:07.404620 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:07.404591 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-c9mnf" Jul 30 08:13:10.911423 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:10.911369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:13:10.913811 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:10.913787 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls\") pod \"image-registry-7b8bbf9d84-drkkr\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:13:10.982714 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:10.982668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:13:11.106397 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:11.106341 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b8bbf9d84-drkkr"] Jul 30 08:13:11.109639 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:13:11.109613 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dccc1d2_4495_4284_8f11_bf0e283dabb9.slice/crio-eeae097ea20c020bde7e8cef10acfe2615a9be89dbde1cec69f40f3cc87010a2 WatchSource:0}: Error finding container eeae097ea20c020bde7e8cef10acfe2615a9be89dbde1cec69f40f3cc87010a2: Status 404 returned error can't find the container with id eeae097ea20c020bde7e8cef10acfe2615a9be89dbde1cec69f40f3cc87010a2 Jul 30 08:13:11.655900 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:11.655866 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" event={"ID":"8dccc1d2-4495-4284-8f11-bf0e283dabb9","Type":"ContainerStarted","Data":"e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a"} Jul 30 08:13:11.656095 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:11.655920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" event={"ID":"8dccc1d2-4495-4284-8f11-bf0e283dabb9","Type":"ContainerStarted","Data":"eeae097ea20c020bde7e8cef10acfe2615a9be89dbde1cec69f40f3cc87010a2"} Jul 30 08:13:11.656095 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:11.656044 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:13:11.673181 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:11.673131 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" podStartSLOduration=16.67311742 podStartE2EDuration="16.67311742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:13:11.673041171 +0000 UTC m=+85.932987516" watchObservedRunningTime="2026-07-30 08:13:11.67311742 +0000 UTC m=+85.933063754" Jul 30 08:13:14.435744 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:14.435704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:13:14.436252 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:14.435894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:13:14.436566 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:14.436542 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b835a639-ac08-499a-9b99-79744d0e8e1b-service-ca-bundle\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:13:14.438088 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:14.438064 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b835a639-ac08-499a-9b99-79744d0e8e1b-metrics-certs\") pod \"router-default-68585dd8ff-bhdlw\" (UID: \"b835a639-ac08-499a-9b99-79744d0e8e1b\") " pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:13:14.735231 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:14.735150 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8w4sn\"" Jul 30 08:13:14.743319 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:14.743293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:13:14.865384 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:14.865337 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68585dd8ff-bhdlw"] Jul 30 08:13:14.869340 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:13:14.869307 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb835a639_ac08_499a_9b99_79744d0e8e1b.slice/crio-c81870ab721a7e14e7824bfc87f2b20d6057217a10363274479daf3c0a4772de WatchSource:0}: Error finding container c81870ab721a7e14e7824bfc87f2b20d6057217a10363274479daf3c0a4772de: Status 404 returned error can't find the container with id c81870ab721a7e14e7824bfc87f2b20d6057217a10363274479daf3c0a4772de Jul 30 08:13:15.669934 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:15.669898 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68585dd8ff-bhdlw" event={"ID":"b835a639-ac08-499a-9b99-79744d0e8e1b","Type":"ContainerStarted","Data":"c8774be356c864be146a9b76e6f3a4bc09156f2a7560114bd47929b0b5242269"} Jul 30 08:13:15.670317 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:15.669952 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68585dd8ff-bhdlw" event={"ID":"b835a639-ac08-499a-9b99-79744d0e8e1b","Type":"ContainerStarted","Data":"c81870ab721a7e14e7824bfc87f2b20d6057217a10363274479daf3c0a4772de"} Jul 30 08:13:15.688642 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:15.688597 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-68585dd8ff-bhdlw" podStartSLOduration=33.688583475 podStartE2EDuration="33.688583475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:13:15.687460924 +0000 UTC m=+89.947407250" watchObservedRunningTime="2026-07-30 08:13:15.688583475 +0000 UTC m=+89.948529810" Jul 30 08:13:15.743708 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:15.743675 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:13:15.746289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:15.746265 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:13:16.673358 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:16.673329 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:13:16.674539 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:16.674519 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-68585dd8ff-bhdlw" Jul 30 08:13:17.415132 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.415092 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh"] Jul 30 08:13:17.418241 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.418217 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh" Jul 30 08:13:17.421717 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.421694 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-skmb8\"" Jul 30 08:13:17.421717 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.421712 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 30 08:13:17.421919 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.421722 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 30 08:13:17.429979 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.429955 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh"] Jul 30 08:13:17.439669 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.439646 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b8bbf9d84-drkkr"] Jul 30 08:13:17.457561 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.457529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/175def52-ca03-46f0-8b82-ee15bb501ee6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-vjbrh\" (UID: \"175def52-ca03-46f0-8b82-ee15bb501ee6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh" Jul 30 08:13:17.457669 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.457653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/175def52-ca03-46f0-8b82-ee15bb501ee6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-vjbrh\" (UID: \"175def52-ca03-46f0-8b82-ee15bb501ee6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh" Jul 30 08:13:17.473342 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.473319 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55b5bd6c4d-6xpbk"] Jul 30 08:13:17.475409 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.475391 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.489552 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.489525 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55b5bd6c4d-6xpbk"] Jul 30 08:13:17.558771 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.558712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9077097e-03b5-409f-a035-9f6a2614bb55-image-registry-private-configuration\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.558976 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.558814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9077097e-03b5-409f-a035-9f6a2614bb55-trusted-ca\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.558976 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.558852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/175def52-ca03-46f0-8b82-ee15bb501ee6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-vjbrh\" (UID: \"175def52-ca03-46f0-8b82-ee15bb501ee6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh" Jul 30 08:13:17.558976 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.558894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9077097e-03b5-409f-a035-9f6a2614bb55-registry-tls\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.559122 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.558976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9077097e-03b5-409f-a035-9f6a2614bb55-installation-pull-secrets\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.559122 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.559019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/175def52-ca03-46f0-8b82-ee15bb501ee6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-vjbrh\" (UID: \"175def52-ca03-46f0-8b82-ee15bb501ee6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh" Jul 30 08:13:17.559122 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.559053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9077097e-03b5-409f-a035-9f6a2614bb55-ca-trust-extracted\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.559122 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.559087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9077097e-03b5-409f-a035-9f6a2614bb55-registry-certificates\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.559297 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.559148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx24j\" (UniqueName: \"kubernetes.io/projected/9077097e-03b5-409f-a035-9f6a2614bb55-kube-api-access-kx24j\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.559297 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.559196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9077097e-03b5-409f-a035-9f6a2614bb55-bound-sa-token\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.563364 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.563335 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/175def52-ca03-46f0-8b82-ee15bb501ee6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-vjbrh\" (UID: \"175def52-ca03-46f0-8b82-ee15bb501ee6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh" Jul 30 08:13:17.565796 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.565769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/175def52-ca03-46f0-8b82-ee15bb501ee6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-vjbrh\" (UID: \"175def52-ca03-46f0-8b82-ee15bb501ee6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh" Jul 30 08:13:17.659556 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.659523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9077097e-03b5-409f-a035-9f6a2614bb55-ca-trust-extracted\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.659556 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.659560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9077097e-03b5-409f-a035-9f6a2614bb55-registry-certificates\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.659769 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.659606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx24j\" (UniqueName: \"kubernetes.io/projected/9077097e-03b5-409f-a035-9f6a2614bb55-kube-api-access-kx24j\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.659769 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.659630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9077097e-03b5-409f-a035-9f6a2614bb55-bound-sa-token\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.659769 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.659685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9077097e-03b5-409f-a035-9f6a2614bb55-image-registry-private-configuration\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.659769 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.659714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9077097e-03b5-409f-a035-9f6a2614bb55-trusted-ca\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.659769 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.659744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9077097e-03b5-409f-a035-9f6a2614bb55-registry-tls\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.659991 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.659797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9077097e-03b5-409f-a035-9f6a2614bb55-installation-pull-secrets\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.660041 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.659990 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9077097e-03b5-409f-a035-9f6a2614bb55-ca-trust-extracted\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.660565 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.660542 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9077097e-03b5-409f-a035-9f6a2614bb55-registry-certificates\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.660674 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.660622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9077097e-03b5-409f-a035-9f6a2614bb55-trusted-ca\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.662220 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.662199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9077097e-03b5-409f-a035-9f6a2614bb55-registry-tls\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.662517 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.662500 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9077097e-03b5-409f-a035-9f6a2614bb55-image-registry-private-configuration\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.662585 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.662557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9077097e-03b5-409f-a035-9f6a2614bb55-installation-pull-secrets\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.669389 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.669316 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx24j\" (UniqueName: \"kubernetes.io/projected/9077097e-03b5-409f-a035-9f6a2614bb55-kube-api-access-kx24j\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.669505 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.669487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9077097e-03b5-409f-a035-9f6a2614bb55-bound-sa-token\") pod \"image-registry-55b5bd6c4d-6xpbk\" (UID: \"9077097e-03b5-409f-a035-9f6a2614bb55\") " pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.727916 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.727875 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh" Jul 30 08:13:17.783868 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.783837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:17.853727 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.853693 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh"] Jul 30 08:13:17.857550 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:13:17.857518 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175def52_ca03_46f0_8b82_ee15bb501ee6.slice/crio-79e5da59555c806a556081c642120b7514d14999b0c0aaa5b8d7c1ab9cf18337 WatchSource:0}: Error finding container 79e5da59555c806a556081c642120b7514d14999b0c0aaa5b8d7c1ab9cf18337: Status 404 returned error can't find the container with id 79e5da59555c806a556081c642120b7514d14999b0c0aaa5b8d7c1ab9cf18337 Jul 30 08:13:17.916643 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:17.916610 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55b5bd6c4d-6xpbk"] Jul 30 08:13:17.921311 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:13:17.921282 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9077097e_03b5_409f_a035_9f6a2614bb55.slice/crio-bd422135f79f0f8b091750255cfa6be8c7ee35a51bc685f4d4d7ac71c1251135 WatchSource:0}: Error finding container bd422135f79f0f8b091750255cfa6be8c7ee35a51bc685f4d4d7ac71c1251135: Status 404 returned error can't find the container with id bd422135f79f0f8b091750255cfa6be8c7ee35a51bc685f4d4d7ac71c1251135 Jul 30 08:13:18.568307 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:18.568269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:13:18.570516 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:18.570487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7069b820-c097-4267-94ed-7c7d5ca4f8d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4twmv\" (UID: \"7069b820-c097-4267-94ed-7c7d5ca4f8d3\") " pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:13:18.680356 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:18.680325 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" event={"ID":"9077097e-03b5-409f-a035-9f6a2614bb55","Type":"ContainerStarted","Data":"259f9fbb8bbb8ca00aa76e7fff8f855f2370291e40edfeda1b24856033b54fdc"} Jul 30 08:13:18.680550 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:18.680387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" event={"ID":"9077097e-03b5-409f-a035-9f6a2614bb55","Type":"ContainerStarted","Data":"bd422135f79f0f8b091750255cfa6be8c7ee35a51bc685f4d4d7ac71c1251135"} Jul 30 08:13:18.680550 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:18.680412 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:18.681620 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:18.681597 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh" event={"ID":"175def52-ca03-46f0-8b82-ee15bb501ee6","Type":"ContainerStarted","Data":"79e5da59555c806a556081c642120b7514d14999b0c0aaa5b8d7c1ab9cf18337"} Jul 30 08:13:18.699187 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:18.699142 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" podStartSLOduration=1.6991274939999998 podStartE2EDuration="1.699127494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:13:18.697807349 +0000 UTC m=+92.957753703" watchObservedRunningTime="2026-07-30 08:13:18.699127494 +0000 UTC m=+92.959073839" Jul 30 08:13:18.808907 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:18.808875 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4twmv" Jul 30 08:13:18.945636 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:18.945599 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4twmv"] Jul 30 08:13:18.948331 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:13:18.948306 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7069b820_c097_4267_94ed_7c7d5ca4f8d3.slice/crio-74f116e29887e0832b1b9d0cd0f44ee79bd3fa785abb7cb5af5951107f340e56 WatchSource:0}: Error finding container 74f116e29887e0832b1b9d0cd0f44ee79bd3fa785abb7cb5af5951107f340e56: Status 404 returned error can't find the container with id 74f116e29887e0832b1b9d0cd0f44ee79bd3fa785abb7cb5af5951107f340e56 Jul 30 08:13:19.689393 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:19.689266 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh" event={"ID":"175def52-ca03-46f0-8b82-ee15bb501ee6","Type":"ContainerStarted","Data":"1d880594e4077eba641658ddd4718aa8514f3abff8518b16fc82bf836fac9723"} Jul 30 08:13:19.691836 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:19.691764 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4twmv" event={"ID":"7069b820-c097-4267-94ed-7c7d5ca4f8d3","Type":"ContainerStarted","Data":"74f116e29887e0832b1b9d0cd0f44ee79bd3fa785abb7cb5af5951107f340e56"} Jul 30 08:13:19.706750 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:19.706497 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-vjbrh" podStartSLOduration=1.168535524 podStartE2EDuration="2.706471628s" totalImagesPullingTime="1.537936104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:13:19.704861844 +0000 UTC m=+93.964808179" watchObservedRunningTime="2026-07-30 08:13:19.706471628 +0000 UTC m=+93.966418023" Jul 30 08:13:22.614544 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.614513 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-h455g"] Jul 30 08:13:22.647369 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.647330 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-h455g"] Jul 30 08:13:22.647533 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.647457 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:22.651834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.651566 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 30 08:13:22.651834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.651700 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 30 08:13:22.651834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.651725 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 30 08:13:22.651834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.651794 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mfxlt\"" Jul 30 08:13:22.652154 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.651892 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 30 08:13:22.652154 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.651947 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 30 08:13:22.707115 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.707076 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fmqb\" (UniqueName: \"kubernetes.io/projected/11949d1b-659e-43d2-9e04-c60edaf4bc45-kube-api-access-4fmqb\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:22.707280 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.707137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11949d1b-659e-43d2-9e04-c60edaf4bc45-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:22.707280 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.707186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/11949d1b-659e-43d2-9e04-c60edaf4bc45-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:22.707280 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.707202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11949d1b-659e-43d2-9e04-c60edaf4bc45-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:22.808047 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.808008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmqb\" (UniqueName: \"kubernetes.io/projected/11949d1b-659e-43d2-9e04-c60edaf4bc45-kube-api-access-4fmqb\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:22.808196 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.808091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11949d1b-659e-43d2-9e04-c60edaf4bc45-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:22.808196 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.808168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/11949d1b-659e-43d2-9e04-c60edaf4bc45-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:22.808328 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.808220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11949d1b-659e-43d2-9e04-c60edaf4bc45-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:22.808328 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:22.808308 2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 30 08:13:22.808437 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:22.808398 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11949d1b-659e-43d2-9e04-c60edaf4bc45-prometheus-operator-tls podName:11949d1b-659e-43d2-9e04-c60edaf4bc45 nodeName:}" failed. No retries permitted until 2026-07-30 08:13:23.308356708 +0000 UTC m=+97.568303028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/11949d1b-659e-43d2-9e04-c60edaf4bc45-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-h455g" (UID: "11949d1b-659e-43d2-9e04-c60edaf4bc45") : secret "prometheus-operator-tls" not found Jul 30 08:13:22.808844 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.808815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11949d1b-659e-43d2-9e04-c60edaf4bc45-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:22.810705 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.810680 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11949d1b-659e-43d2-9e04-c60edaf4bc45-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:22.816470 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:22.816445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fmqb\" (UniqueName: \"kubernetes.io/projected/11949d1b-659e-43d2-9e04-c60edaf4bc45-kube-api-access-4fmqb\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:23.312420 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:23.312363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/11949d1b-659e-43d2-9e04-c60edaf4bc45-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:23.315228 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:23.315197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/11949d1b-659e-43d2-9e04-c60edaf4bc45-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-h455g\" (UID: \"11949d1b-659e-43d2-9e04-c60edaf4bc45\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:23.514953 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:23.514908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:13:23.517274 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:23.517251 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fa47875-28d6-4eb9-86a1-1f9747488d9c-metrics-tls\") pod \"dns-default-qfcjq\" (UID: \"4fa47875-28d6-4eb9-86a1-1f9747488d9c\") " pod="openshift-dns/dns-default-qfcjq" Jul 30 08:13:23.557828 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:23.557796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" Jul 30 08:13:23.795540 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:23.795509 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rrdcg\"" Jul 30 08:13:23.804068 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:23.804047 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qfcjq" Jul 30 08:13:23.818432 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:23.818405 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-h455g"] Jul 30 08:13:23.821744 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:13:23.821705 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11949d1b_659e_43d2_9e04_c60edaf4bc45.slice/crio-64d9d59776a389dbc13ceb914b341c0eabda57d9f9711dd0d24e0bb5923c289f WatchSource:0}: Error finding container 64d9d59776a389dbc13ceb914b341c0eabda57d9f9711dd0d24e0bb5923c289f: Status 404 returned error can't find the container with id 64d9d59776a389dbc13ceb914b341c0eabda57d9f9711dd0d24e0bb5923c289f Jul 30 08:13:23.932993 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:23.932903 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qfcjq"] Jul 30 08:13:23.935428 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:13:23.935400 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa47875_28d6_4eb9_86a1_1f9747488d9c.slice/crio-afc046e4d62b41df5cde203ecff58625c0baa99f8b6208109a992abd49bca9db WatchSource:0}: Error finding container afc046e4d62b41df5cde203ecff58625c0baa99f8b6208109a992abd49bca9db: Status 404 returned error can't find the container with id afc046e4d62b41df5cde203ecff58625c0baa99f8b6208109a992abd49bca9db Jul 30 08:13:24.714258 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:24.714178 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qfcjq" event={"ID":"4fa47875-28d6-4eb9-86a1-1f9747488d9c","Type":"ContainerStarted","Data":"afc046e4d62b41df5cde203ecff58625c0baa99f8b6208109a992abd49bca9db"} Jul 30 08:13:24.715859 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:24.715773 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" event={"ID":"11949d1b-659e-43d2-9e04-c60edaf4bc45","Type":"ContainerStarted","Data":"64d9d59776a389dbc13ceb914b341c0eabda57d9f9711dd0d24e0bb5923c289f"} Jul 30 08:13:24.718586 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:24.718560 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4twmv" event={"ID":"7069b820-c097-4267-94ed-7c7d5ca4f8d3","Type":"ContainerStarted","Data":"b3d55398ccd99e43e1d92d3ec2a75054fc0aec43bbe7e545193045bc687c4cf1"} Jul 30 08:13:27.447007 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:27.446890 2576 patch_prober.go:28] interesting pod/image-registry-7b8bbf9d84-drkkr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 30 08:13:27.447007 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:27.446966 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" podUID="8dccc1d2-4495-4284-8f11-bf0e283dabb9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 08:13:28.616673 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:28.616647 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xp7kt" Jul 30 08:13:28.734287 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:28.734245 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4twmv" event={"ID":"7069b820-c097-4267-94ed-7c7d5ca4f8d3","Type":"ContainerStarted","Data":"3c795c55ef519529775ba1cebdd9a6abf198e2c89ae86c7e8f65d990b6408c4d"} Jul 30 08:13:28.735876 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:28.735847 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qfcjq" event={"ID":"4fa47875-28d6-4eb9-86a1-1f9747488d9c","Type":"ContainerStarted","Data":"b0e5a7a700837e923cbe2d012a949c5f5200b003cae87943508ef700dd40806c"} Jul 30 08:13:28.736000 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:28.735891 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qfcjq" event={"ID":"4fa47875-28d6-4eb9-86a1-1f9747488d9c","Type":"ContainerStarted","Data":"c6b4b3dc6f3d033cc26a7653a130ca63915e65c7478019da12706ca7298f238f"} Jul 30 08:13:28.736000 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:28.735964 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qfcjq" Jul 30 08:13:28.737663 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:28.737646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" event={"ID":"11949d1b-659e-43d2-9e04-c60edaf4bc45","Type":"ContainerStarted","Data":"b8b29d085f107c278358e3d1057c3507b130ca72d1e5b0ee0967007f6f1cc820"} Jul 30 08:13:28.737726 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:28.737679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" event={"ID":"11949d1b-659e-43d2-9e04-c60edaf4bc45","Type":"ContainerStarted","Data":"73794cb2abf3e4c8f26f54764562a1b4c12d7466dcf947e019cceb3e1ae161fc"} Jul 30 08:13:28.753087 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:28.753048 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qfcjq" podStartSLOduration=65.712448906 podStartE2EDuration="1m9.753036254s" totalImagesPullingTime="4.040587348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:13:28.75250549 +0000 UTC m=+103.012451825" watchObservedRunningTime="2026-07-30 08:13:28.753036254 +0000 UTC m=+103.012982591" Jul 30 08:13:28.768042 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:28.768002 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-h455g" podStartSLOduration=2.613883632 podStartE2EDuration="6.767990068s" totalImagesPullingTime="4.154106436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:13:28.767492424 +0000 UTC m=+103.027438757" watchObservedRunningTime="2026-07-30 08:13:28.767990068 +0000 UTC m=+103.027936402" Jul 30 08:13:30.983360 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:30.983326 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx"] Jul 30 08:13:31.081164 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.081133 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lp6qh"] Jul 30 08:13:31.081326 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.081216 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.083815 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.083791 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 30 08:13:31.084942 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.084908 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sv567\"" Jul 30 08:13:31.084942 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.084926 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 30 08:13:31.085108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.084976 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 30 08:13:31.110832 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.110799 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx"] Jul 30 08:13:31.110981 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.110966 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.113408 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.113315 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 30 08:13:31.113408 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.113324 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 30 08:13:31.113408 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.113348 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9v6f4\"" Jul 30 08:13:31.113595 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.113466 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 30 08:13:31.187001 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.186950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-tls\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.187153 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.187153 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187035 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.187153 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a98b277-2607-4cac-bfe3-5023f1b78468-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.187153 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-wtmp\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.187326 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvwms\" (UniqueName: \"kubernetes.io/projected/7e8fc961-8af7-4963-8646-e2820bf06413-kube-api-access-lvwms\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.187326 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-textfile\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.187326 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e8fc961-8af7-4963-8646-e2820bf06413-root\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.187468 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e8fc961-8af7-4963-8646-e2820bf06413-metrics-client-ca\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.187468 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwsz6\" (UniqueName: \"kubernetes.io/projected/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-api-access-rwsz6\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.187468 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.187593 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9a98b277-2607-4cac-bfe3-5023f1b78468-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.187593 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187522 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.187593 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-accelerators-collector-config\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.187718 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.187607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e8fc961-8af7-4963-8646-e2820bf06413-sys\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.288741 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.288715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e8fc961-8af7-4963-8646-e2820bf06413-metrics-client-ca\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.288918 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.288770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwsz6\" (UniqueName: \"kubernetes.io/projected/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-api-access-rwsz6\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.288918 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.288902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.289024 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.288950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9a98b277-2607-4cac-bfe3-5023f1b78468-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.289024 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.288989 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.289164 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:31.289022 2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 30 08:13:31.289164 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-accelerators-collector-config\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289164 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e8fc961-8af7-4963-8646-e2820bf06413-sys\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289164 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:31.289116 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-tls podName:9a98b277-2607-4cac-bfe3-5023f1b78468 nodeName:}" failed. No retries permitted until 2026-07-30 08:13:31.78908145 +0000 UTC m=+106.049027778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-stdlx" (UID: "9a98b277-2607-4cac-bfe3-5023f1b78468") : secret "kube-state-metrics-tls" not found Jul 30 08:13:31.289164 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289128 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e8fc961-8af7-4963-8646-e2820bf06413-sys\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289164 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-tls\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289488 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.289488 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289488 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a98b277-2607-4cac-bfe3-5023f1b78468-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.289488 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-wtmp\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289488 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvwms\" (UniqueName: \"kubernetes.io/projected/7e8fc961-8af7-4963-8646-e2820bf06413-kube-api-access-lvwms\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289488 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-textfile\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289488 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e8fc961-8af7-4963-8646-e2820bf06413-root\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289813 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289509 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e8fc961-8af7-4963-8646-e2820bf06413-root\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289813 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289638 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-accelerators-collector-config\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289813 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289655 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-wtmp\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.289813 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289700 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9a98b277-2607-4cac-bfe3-5023f1b78468-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.289813 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:31.289785 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 30 08:13:31.290039 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:31.289839 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-tls podName:7e8fc961-8af7-4963-8646-e2820bf06413 nodeName:}" failed. No retries permitted until 2026-07-30 08:13:31.789823828 +0000 UTC m=+106.049770158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-tls") pod "node-exporter-lp6qh" (UID: "7e8fc961-8af7-4963-8646-e2820bf06413") : secret "node-exporter-tls" not found Jul 30 08:13:31.290039 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e8fc961-8af7-4963-8646-e2820bf06413-metrics-client-ca\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.290039 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.289940 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-textfile\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.290342 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.290293 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a98b277-2607-4cac-bfe3-5023f1b78468-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.290490 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.290412 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.291868 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.291840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.292068 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.292041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.301409 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.301360 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvwms\" (UniqueName: \"kubernetes.io/projected/7e8fc961-8af7-4963-8646-e2820bf06413-kube-api-access-lvwms\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.301852 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.301834 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwsz6\" (UniqueName: \"kubernetes.io/projected/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-api-access-rwsz6\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.795618 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.795140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.795618 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.795214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-tls\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.797685 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.797658 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e8fc961-8af7-4963-8646-e2820bf06413-node-exporter-tls\") pod \"node-exporter-lp6qh\" (UID: \"7e8fc961-8af7-4963-8646-e2820bf06413\") " pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:31.798020 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.798000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a98b277-2607-4cac-bfe3-5023f1b78468-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-stdlx\" (UID: \"9a98b277-2607-4cac-bfe3-5023f1b78468\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:31.991847 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:31.991816 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" Jul 30 08:13:32.021166 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:32.021142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lp6qh" Jul 30 08:13:32.031854 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:13:32.031828 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8fc961_8af7_4963_8646_e2820bf06413.slice/crio-4652c2d610519989c3b949e8d207e6b9dc4412f82448d9ea3646f9c46c3779a1 WatchSource:0}: Error finding container 4652c2d610519989c3b949e8d207e6b9dc4412f82448d9ea3646f9c46c3779a1: Status 404 returned error can't find the container with id 4652c2d610519989c3b949e8d207e6b9dc4412f82448d9ea3646f9c46c3779a1 Jul 30 08:13:32.134650 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:32.134620 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx"] Jul 30 08:13:32.137246 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:13:32.137220 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a98b277_2607_4cac_bfe3_5023f1b78468.slice/crio-55b2bb65e48f830e2595886ea230bac52a6aa8824a6962e3699449941a14a7c5 WatchSource:0}: Error finding container 55b2bb65e48f830e2595886ea230bac52a6aa8824a6962e3699449941a14a7c5: Status 404 returned error can't find the container with id 55b2bb65e48f830e2595886ea230bac52a6aa8824a6962e3699449941a14a7c5 Jul 30 08:13:32.754078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:32.754017 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4twmv" event={"ID":"7069b820-c097-4267-94ed-7c7d5ca4f8d3","Type":"ContainerStarted","Data":"434e940656b11c4eaaaf80f13a0e0c15e49067069cd9d9fcf2804b48587f1c9c"} Jul 30 08:13:32.755180 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:32.755154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" event={"ID":"9a98b277-2607-4cac-bfe3-5023f1b78468","Type":"ContainerStarted","Data":"55b2bb65e48f830e2595886ea230bac52a6aa8824a6962e3699449941a14a7c5"} Jul 30 08:13:32.756311 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:32.756281 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lp6qh" event={"ID":"7e8fc961-8af7-4963-8646-e2820bf06413","Type":"ContainerStarted","Data":"4652c2d610519989c3b949e8d207e6b9dc4412f82448d9ea3646f9c46c3779a1"} Jul 30 08:13:32.772653 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:32.772613 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4twmv" podStartSLOduration=33.905521284 podStartE2EDuration="46.772597675s" totalImagesPullingTime="12.867076391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:12:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:13:32.771074228 +0000 UTC m=+107.031020574" watchObservedRunningTime="2026-07-30 08:13:32.772597675 +0000 UTC m=+107.032544009" Jul 30 08:13:34.765192 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:34.765158 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e8fc961-8af7-4963-8646-e2820bf06413" containerID="59198c44bf72f69de37c85ede25ae52d3a822abdf0ccc61f92ca91f52011bec1" exitCode=0 Jul 30 08:13:34.765614 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:34.765241 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lp6qh" event={"ID":"7e8fc961-8af7-4963-8646-e2820bf06413","Type":"ContainerDied","Data":"59198c44bf72f69de37c85ede25ae52d3a822abdf0ccc61f92ca91f52011bec1"} Jul 30 08:13:34.767363 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:34.767339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" event={"ID":"9a98b277-2607-4cac-bfe3-5023f1b78468","Type":"ContainerStarted","Data":"859c89f0394558fb0e93fbba89895186e86d13b0b3e08276aa3962128b0a8d09"} Jul 30 08:13:34.767494 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:34.767397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" event={"ID":"9a98b277-2607-4cac-bfe3-5023f1b78468","Type":"ContainerStarted","Data":"251a929a20190e527dfca71ce0ca88920195a7fc45ef56fef6d7d5fd5200576b"} Jul 30 08:13:34.767494 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:34.767428 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" event={"ID":"9a98b277-2607-4cac-bfe3-5023f1b78468","Type":"ContainerStarted","Data":"8fe23992522fb934056dc0a401cb4af9babc7a3bc9dd0ce1343b3885973c94bb"} Jul 30 08:13:34.801360 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:34.801316 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stdlx" podStartSLOduration=3.086239345 podStartE2EDuration="4.801301279s" totalImagesPullingTime="1.715061934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:13:34.800634226 +0000 UTC m=+109.060580577" watchObservedRunningTime="2026-07-30 08:13:34.801301279 +0000 UTC m=+109.061247614" Jul 30 08:13:35.773747 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:35.773708 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lp6qh" event={"ID":"7e8fc961-8af7-4963-8646-e2820bf06413","Type":"ContainerStarted","Data":"80888efa95d31333571d4caa29b92dbc827bcd7781395311bbd9fc4929d00085"} Jul 30 08:13:35.774125 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:35.773774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lp6qh" event={"ID":"7e8fc961-8af7-4963-8646-e2820bf06413","Type":"ContainerStarted","Data":"88e907912132bfb98e67632cc7ff43722a68f6f271441ac15d49e3795fff0564"} Jul 30 08:13:35.799069 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:35.799022 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lp6qh" podStartSLOduration=3.896262628 podStartE2EDuration="5.79899137s" totalImagesPullingTime="1.818483753s" totalInitContainerRuntime="84.244989ms" isStatefulPod=true podCreationTimestamp="2026-07-30 08:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:13:35.797529783 +0000 UTC m=+110.057476119" watchObservedRunningTime="2026-07-30 08:13:35.79899137 +0000 UTC m=+110.058937763" Jul 30 08:13:37.444796 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:37.444765 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:13:38.744185 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:38.744156 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qfcjq" Jul 30 08:13:39.696413 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:39.696363 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55b5bd6c4d-6xpbk" Jul 30 08:13:42.459249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.459207 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" podUID="8dccc1d2-4495-4284-8f11-bf0e283dabb9" containerName="registry" containerID="cri-o://e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a" gracePeriod=30 Jul 30 08:13:42.689883 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.689861 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:13:42.791539 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.791512 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8dccc1d2-4495-4284-8f11-bf0e283dabb9-ca-trust-extracted\") pod \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " Jul 30 08:13:42.791730 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.791555 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-certificates\") pod \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " Jul 30 08:13:42.791730 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.791581 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls\") pod \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " Jul 30 08:13:42.791730 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.791624 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-bound-sa-token\") pod \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " Jul 30 08:13:42.791730 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.791648 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-image-registry-private-configuration\") pod \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " Jul 30 08:13:42.791730 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.791699 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-installation-pull-secrets\") pod \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " Jul 30 08:13:42.791978 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.791744 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs8gg\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-kube-api-access-cs8gg\") pod \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " Jul 30 08:13:42.791978 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.791770 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-trusted-ca\") pod \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\" (UID: \"8dccc1d2-4495-4284-8f11-bf0e283dabb9\") " Jul 30 08:13:42.792287 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.792150 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8dccc1d2-4495-4284-8f11-bf0e283dabb9" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 08:13:42.792398 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.792361 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8dccc1d2-4495-4284-8f11-bf0e283dabb9" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 08:13:42.794354 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.794325 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8dccc1d2-4495-4284-8f11-bf0e283dabb9" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:13:42.794485 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.794348 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-kube-api-access-cs8gg" (OuterVolumeSpecName: "kube-api-access-cs8gg") pod "8dccc1d2-4495-4284-8f11-bf0e283dabb9" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9"). InnerVolumeSpecName "kube-api-access-cs8gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:13:42.794485 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.794456 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8dccc1d2-4495-4284-8f11-bf0e283dabb9" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:13:42.794485 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.794457 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8dccc1d2-4495-4284-8f11-bf0e283dabb9" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:13:42.794597 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.794477 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8dccc1d2-4495-4284-8f11-bf0e283dabb9" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:13:42.799615 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.799591 2576 generic.go:356] "Generic (PLEG): container finished" podID="8dccc1d2-4495-4284-8f11-bf0e283dabb9" containerID="e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a" exitCode=0 Jul 30 08:13:42.799709 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.799647 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" Jul 30 08:13:42.799709 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.799661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" event={"ID":"8dccc1d2-4495-4284-8f11-bf0e283dabb9","Type":"ContainerDied","Data":"e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a"} Jul 30 08:13:42.799780 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.799729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b8bbf9d84-drkkr" event={"ID":"8dccc1d2-4495-4284-8f11-bf0e283dabb9","Type":"ContainerDied","Data":"eeae097ea20c020bde7e8cef10acfe2615a9be89dbde1cec69f40f3cc87010a2"} Jul 30 08:13:42.799780 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.799752 2576 scope.go:117] "RemoveContainer" containerID="e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a" Jul 30 08:13:42.802253 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.802222 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dccc1d2-4495-4284-8f11-bf0e283dabb9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8dccc1d2-4495-4284-8f11-bf0e283dabb9" (UID: "8dccc1d2-4495-4284-8f11-bf0e283dabb9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:13:42.811107 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.811091 2576 scope.go:117] "RemoveContainer" containerID="e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a" Jul 30 08:13:42.811392 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:13:42.811359 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a\": container with ID starting with e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a not found: ID does not exist" containerID="e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a" Jul 30 08:13:42.811456 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.811403 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a"} err="failed to get container status \"e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a\": rpc error: code = NotFound desc = could not find container \"e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a\": container with ID starting with e5a895ee5572240e2274a935568c51e568fb6e1e5136f36935fb1c8205067e4a not found: ID does not exist" Jul 30 08:13:42.892633 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.892599 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-bound-sa-token\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:13:42.892633 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.892631 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-image-registry-private-configuration\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:13:42.892787 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.892645 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8dccc1d2-4495-4284-8f11-bf0e283dabb9-installation-pull-secrets\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:13:42.892787 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.892658 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs8gg\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-kube-api-access-cs8gg\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:13:42.892787 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.892670 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-trusted-ca\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:13:42.892787 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.892681 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8dccc1d2-4495-4284-8f11-bf0e283dabb9-ca-trust-extracted\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:13:42.892787 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.892692 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-certificates\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:13:42.892787 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:42.892703 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dccc1d2-4495-4284-8f11-bf0e283dabb9-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:13:43.120345 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:43.120314 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b8bbf9d84-drkkr"] Jul 30 08:13:43.127152 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:43.127126 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b8bbf9d84-drkkr"] Jul 30 08:13:44.291722 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:44.291683 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dccc1d2-4495-4284-8f11-bf0e283dabb9" path="/var/lib/kubelet/pods/8dccc1d2-4495-4284-8f11-bf0e283dabb9/volumes" Jul 30 08:13:56.107641 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:56.107602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:13:56.109802 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:56.109772 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f661b99d-6531-403f-8622-dec42ada1174-metrics-certs\") pod \"network-metrics-daemon-rkhqs\" (UID: \"f661b99d-6531-403f-8622-dec42ada1174\") " pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:13:56.409840 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:56.409763 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t6dmg\"" Jul 30 08:13:56.417176 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:56.417156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkhqs" Jul 30 08:13:56.565872 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:56.565843 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rkhqs"] Jul 30 08:13:56.568820 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:13:56.568794 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf661b99d_6531_403f_8622_dec42ada1174.slice/crio-9ada8c2464bc7f6cb251874436f7b5ceeb023b6e6d79ea621917820fab3c1e96 WatchSource:0}: Error finding container 9ada8c2464bc7f6cb251874436f7b5ceeb023b6e6d79ea621917820fab3c1e96: Status 404 returned error can't find the container with id 9ada8c2464bc7f6cb251874436f7b5ceeb023b6e6d79ea621917820fab3c1e96 Jul 30 08:13:56.850685 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:56.850646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkhqs" event={"ID":"f661b99d-6531-403f-8622-dec42ada1174","Type":"ContainerStarted","Data":"9ada8c2464bc7f6cb251874436f7b5ceeb023b6e6d79ea621917820fab3c1e96"} Jul 30 08:13:57.855742 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:57.855703 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkhqs" event={"ID":"f661b99d-6531-403f-8622-dec42ada1174","Type":"ContainerStarted","Data":"e4000b274e9e09180196485ba2d021a0eaa83149455d5b604c206d4f2ea51157"} Jul 30 08:13:57.856230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:57.855765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkhqs" event={"ID":"f661b99d-6531-403f-8622-dec42ada1174","Type":"ContainerStarted","Data":"4186bc31393d74adf288a7ef9e3c9309c96e0a83f28b99b92eced18b90118b7e"} Jul 30 08:13:57.872419 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:13:57.872350 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rkhqs" podStartSLOduration=131.097532398 podStartE2EDuration="2m11.872333799s" totalImagesPullingTime="774.801401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:13:57.871002959 +0000 UTC m=+132.130949295" watchObservedRunningTime="2026-07-30 08:13:57.872333799 +0000 UTC m=+132.132280133" Jul 30 08:14:16.922772 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:14:16.922737 2576 generic.go:356] "Generic (PLEG): container finished" podID="0ed27260-2f69-4fba-8783-e7155daa29e5" containerID="aeacc1b86ab47c148cf5c8b22fa95c1bed491e8b413e9a10ba71abc98df3044b" exitCode=0 Jul 30 08:14:16.923198 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:14:16.922809 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-75lsm" event={"ID":"0ed27260-2f69-4fba-8783-e7155daa29e5","Type":"ContainerDied","Data":"aeacc1b86ab47c148cf5c8b22fa95c1bed491e8b413e9a10ba71abc98df3044b"} Jul 30 08:14:16.923198 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:14:16.923169 2576 scope.go:117] "RemoveContainer" containerID="aeacc1b86ab47c148cf5c8b22fa95c1bed491e8b413e9a10ba71abc98df3044b" Jul 30 08:14:17.932035 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:14:17.932001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-75lsm" event={"ID":"0ed27260-2f69-4fba-8783-e7155daa29e5","Type":"ContainerStarted","Data":"39721ffc3d67ecb065a193d25462f247f52acfc36ac60b2d2ae1c7d90844dee3"} Jul 30 08:16:40.835795 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.835758 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s"] Jul 30 08:16:40.836206 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.836030 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dccc1d2-4495-4284-8f11-bf0e283dabb9" containerName="registry" Jul 30 08:16:40.836206 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.836040 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dccc1d2-4495-4284-8f11-bf0e283dabb9" containerName="registry" Jul 30 08:16:40.836206 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.836092 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dccc1d2-4495-4284-8f11-bf0e283dabb9" containerName="registry" Jul 30 08:16:40.838729 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.838713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" Jul 30 08:16:40.843584 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.843553 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jul 30 08:16:40.843725 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.843711 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jul 30 08:16:40.844946 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.844796 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-vx6bj\"" Jul 30 08:16:40.853773 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.853738 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s"] Jul 30 08:16:40.866593 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.866559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vp9\" (UniqueName: \"kubernetes.io/projected/88463ebb-d6e7-4acf-8bb8-51e81bd1ee66-kube-api-access-r7vp9\") pod \"feast-operator-controller-manager-67978c95f8-5sp7s\" (UID: \"88463ebb-d6e7-4acf-8bb8-51e81bd1ee66\") " pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" Jul 30 08:16:40.967946 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.967910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vp9\" (UniqueName: \"kubernetes.io/projected/88463ebb-d6e7-4acf-8bb8-51e81bd1ee66-kube-api-access-r7vp9\") pod \"feast-operator-controller-manager-67978c95f8-5sp7s\" (UID: \"88463ebb-d6e7-4acf-8bb8-51e81bd1ee66\") " pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" Jul 30 08:16:40.977291 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:40.977257 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vp9\" (UniqueName: \"kubernetes.io/projected/88463ebb-d6e7-4acf-8bb8-51e81bd1ee66-kube-api-access-r7vp9\") pod \"feast-operator-controller-manager-67978c95f8-5sp7s\" (UID: \"88463ebb-d6e7-4acf-8bb8-51e81bd1ee66\") " pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" Jul 30 08:16:41.148056 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:41.147974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" Jul 30 08:16:41.293457 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:41.293427 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s"] Jul 30 08:16:41.295875 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:16:41.295846 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88463ebb_d6e7_4acf_8bb8_51e81bd1ee66.slice/crio-950a95cb0f022bce3163180c936d0a1213acb71d4bcfa73c60f3154435cb809f WatchSource:0}: Error finding container 950a95cb0f022bce3163180c936d0a1213acb71d4bcfa73c60f3154435cb809f: Status 404 returned error can't find the container with id 950a95cb0f022bce3163180c936d0a1213acb71d4bcfa73c60f3154435cb809f Jul 30 08:16:41.427440 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:41.427332 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" event={"ID":"88463ebb-d6e7-4acf-8bb8-51e81bd1ee66","Type":"ContainerStarted","Data":"950a95cb0f022bce3163180c936d0a1213acb71d4bcfa73c60f3154435cb809f"} Jul 30 08:16:44.440135 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:44.440099 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" event={"ID":"88463ebb-d6e7-4acf-8bb8-51e81bd1ee66","Type":"ContainerStarted","Data":"f7810a97da7e5a4ad886fd1f7206ecf60e7bcf6e302efcc30b06b40e2982cc95"} Jul 30 08:16:44.440547 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:44.440229 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" Jul 30 08:16:44.465059 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:44.465010 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" podStartSLOduration=1.844176313 podStartE2EDuration="4.464990202s" totalImagesPullingTime="2.620813889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:16:44.463964482 +0000 UTC m=+298.723910817" watchObservedRunningTime="2026-07-30 08:16:44.464990202 +0000 UTC m=+298.724936537" Jul 30 08:16:46.120034 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:46.120011 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:16:46.126531 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:46.126507 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:16:46.127187 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:46.127169 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:16:46.130000 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:46.129981 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 30 08:16:46.134444 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:46.134421 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:16:55.446152 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:16:55.446120 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" Jul 30 08:17:18.821689 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.821650 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv"] Jul 30 08:17:18.826901 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.826872 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:18.829422 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.829397 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 30 08:17:18.829578 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.829399 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 30 08:17:18.829578 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.829478 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-9xppz\"" Jul 30 08:17:18.829695 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.829612 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 30 08:17:18.829801 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.829785 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 30 08:17:18.829916 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.829894 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 30 08:17:18.830355 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.830337 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 30 08:17:18.830442 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.830418 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 30 08:17:18.838154 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.838129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv"] Jul 30 08:17:18.978673 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.978629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-offline-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:18.978854 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.978679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-registry-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:18.978854 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.978707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59444af2-8a57-4e5c-9bdd-8c727baa576e-feast-data\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:18.978854 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.978759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-online-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:18.978854 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.978839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-ui-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:18.978999 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:18.978861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v758\" (UniqueName: \"kubernetes.io/projected/59444af2-8a57-4e5c-9bdd-8c727baa576e-kube-api-access-9v758\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.080332 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.080238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-offline-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.080332 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.080280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-registry-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.080332 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.080297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59444af2-8a57-4e5c-9bdd-8c727baa576e-feast-data\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.080664 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:17:19.080419 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:17:19.080664 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.080438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-online-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.080664 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:17:19.080441 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 30 08:17:19.080664 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:17:19.080482 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-offline-tls podName:59444af2-8a57-4e5c-9bdd-8c727baa576e nodeName:}" failed. No retries permitted until 2026-07-30 08:17:19.580464176 +0000 UTC m=+333.840410492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-offline-tls") pod "feast-simple-feast-setup-554b65bbfb-9szhv" (UID: "59444af2-8a57-4e5c-9bdd-8c727baa576e") : secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:17:19.080664 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:17:19.080566 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-registry-tls podName:59444af2-8a57-4e5c-9bdd-8c727baa576e nodeName:}" failed. No retries permitted until 2026-07-30 08:17:19.580550434 +0000 UTC m=+333.840496759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-registry-tls") pod "feast-simple-feast-setup-554b65bbfb-9szhv" (UID: "59444af2-8a57-4e5c-9bdd-8c727baa576e") : secret "feast-simple-feast-setup-registry-tls" not found Jul 30 08:17:19.080664 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.080581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-ui-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.080664 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.080605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v758\" (UniqueName: \"kubernetes.io/projected/59444af2-8a57-4e5c-9bdd-8c727baa576e-kube-api-access-9v758\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.080945 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.080699 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59444af2-8a57-4e5c-9bdd-8c727baa576e-feast-data\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.082962 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.082937 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-ui-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.083094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.082937 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-online-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.093125 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.093093 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v758\" (UniqueName: \"kubernetes.io/projected/59444af2-8a57-4e5c-9bdd-8c727baa576e-kube-api-access-9v758\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.584703 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.584668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-offline-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.584703 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.584710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-registry-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.587098 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.587072 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-registry-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.587213 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.587104 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-offline-tls\") pod \"feast-simple-feast-setup-554b65bbfb-9szhv\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.743080 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.743037 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:17:19.911321 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.911292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv"] Jul 30 08:17:19.912863 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:17:19.912836 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59444af2_8a57_4e5c_9bdd_8c727baa576e.slice/crio-912eaa4f0cab7b1dd32590a6a4008ef495f761975c3a4a24ef7138a771010861 WatchSource:0}: Error finding container 912eaa4f0cab7b1dd32590a6a4008ef495f761975c3a4a24ef7138a771010861: Status 404 returned error can't find the container with id 912eaa4f0cab7b1dd32590a6a4008ef495f761975c3a4a24ef7138a771010861 Jul 30 08:17:19.914700 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:19.914685 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 08:17:20.569645 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:20.569606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerStarted","Data":"912eaa4f0cab7b1dd32590a6a4008ef495f761975c3a4a24ef7138a771010861"} Jul 30 08:17:48.678070 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:48.678028 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerStarted","Data":"084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a"} Jul 30 08:17:53.699002 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:53.698968 2576 generic.go:356] "Generic (PLEG): container finished" podID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerID="084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a" exitCode=0 Jul 30 08:17:53.699451 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:53.699008 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerDied","Data":"084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a"} Jul 30 08:17:54.705188 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:54.705152 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerStarted","Data":"4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48"} Jul 30 08:17:59.726675 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:59.726584 2576 generic.go:356] "Generic (PLEG): container finished" podID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerID="4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48" exitCode=0 Jul 30 08:17:59.726675 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:59.726656 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerDied","Data":"4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48"} Jul 30 08:17:59.736971 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:17:59.736953 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 30 08:18:00.736633 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:00.736587 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerStarted","Data":"a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d"} Jul 30 08:18:00.737111 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:00.736657 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerStarted","Data":"a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55"} Jul 30 08:18:00.737111 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:00.736680 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerStarted","Data":"b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3"} Jul 30 08:18:00.737111 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:00.736699 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerStarted","Data":"7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e"} Jul 30 08:18:00.763915 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:00.763850 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podStartSLOduration=14.454742981 podStartE2EDuration="42.763826431s" totalImagesPullingTime="28.30908345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:18:00.762641855 +0000 UTC m=+375.022588216" watchObservedRunningTime="2026-07-30 08:18:00.763826431 +0000 UTC m=+375.023772767" Jul 30 08:18:01.744552 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:01.743549 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:01.744552 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:01.743609 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:01.744552 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:01.743630 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:01.744552 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:01.743650 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:01.745709 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:01.745658 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.19:8443: connect: connection refused" Jul 30 08:18:01.745838 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:01.745698 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.19:6571: connect: connection refused" Jul 30 08:18:01.745838 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:01.745745 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.19:8816: connect: connection refused" Jul 30 08:18:01.745838 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:01.745698 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.19:6567/health\": dial tcp 10.134.0.19:6567: connect: connection refused" Jul 30 08:18:04.743705 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:04.743662 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.19:8443: connect: connection refused" Jul 30 08:18:04.744342 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:04.743670 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.19:6567/health\": dial tcp 10.134.0.19:6567: connect: connection refused" Jul 30 08:18:04.744342 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:04.743667 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.19:8816: connect: connection refused" Jul 30 08:18:04.744342 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:04.743674 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.19:6571: connect: connection refused" Jul 30 08:18:07.744437 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.744406 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:07.745212 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.745188 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:07.745401 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.745389 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:07.745828 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.745797 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:07.745942 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.745849 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:07.745942 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.745867 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:07.746051 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.746013 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:07.746092 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.746079 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:07.746226 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.746213 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:07.750089 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.750074 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:07.770337 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.770311 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:18:07.773404 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:18:07.773385 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:19:10.159113 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.159016 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv"] Jul 30 08:19:10.159693 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.159290 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="registry" containerID="cri-o://7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e" gracePeriod=30 Jul 30 08:19:10.159693 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.159362 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="ui" containerID="cri-o://a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d" gracePeriod=30 Jul 30 08:19:10.159693 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.159474 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="offline" containerID="cri-o://a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55" gracePeriod=30 Jul 30 08:19:10.159693 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.159548 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="online" containerID="cri-o://b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3" gracePeriod=30 Jul 30 08:19:10.447336 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.447252 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh"] Jul 30 08:19:10.451138 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.451112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.453720 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.453700 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-wtfdg\"" Jul 30 08:19:10.465201 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.465171 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh"] Jul 30 08:19:10.587295 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.587249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-registry-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.587497 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.587304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.587497 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.587429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ptl\" (UniqueName: \"kubernetes.io/projected/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-kube-api-access-b7ptl\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.587574 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.587493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-online-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.587574 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.587523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-ui-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.587574 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.587549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-feast-data\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.688785 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.688752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-online-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.688974 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.688829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-ui-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.688974 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.688865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-feast-data\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.688974 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.688902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-registry-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.688974 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.688931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.689177 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.689026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ptl\" (UniqueName: \"kubernetes.io/projected/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-kube-api-access-b7ptl\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.689177 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:10.689041 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 30 08:19:10.689177 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:10.689124 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:19:10.689177 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:10.689156 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-registry-tls podName:efe7bb09-143e-4cc6-b7ef-b2446ff5fb35 nodeName:}" failed. No retries permitted until 2026-07-30 08:19:11.189132577 +0000 UTC m=+445.449078912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-registry-tls") pod "feast-simple-feast-setup-554b65bbfb-l8wfh" (UID: "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35") : secret "feast-simple-feast-setup-registry-tls" not found Jul 30 08:19:10.689411 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:10.689199 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls podName:efe7bb09-143e-4cc6-b7ef-b2446ff5fb35 nodeName:}" failed. No retries permitted until 2026-07-30 08:19:11.189180275 +0000 UTC m=+445.449126590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls") pod "feast-simple-feast-setup-554b65bbfb-l8wfh" (UID: "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35") : secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:19:10.689579 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.689559 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-feast-data\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.691643 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.691609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-online-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.691857 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.691835 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-ui-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:10.697648 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:10.697578 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ptl\" (UniqueName: \"kubernetes.io/projected/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-kube-api-access-b7ptl\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:11.003259 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:11.003173 2576 generic.go:356] "Generic (PLEG): container finished" podID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerID="a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d" exitCode=0 Jul 30 08:19:11.003259 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:11.003197 2576 generic.go:356] "Generic (PLEG): container finished" podID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerID="b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3" exitCode=0 Jul 30 08:19:11.003259 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:11.003246 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerDied","Data":"a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d"} Jul 30 08:19:11.003494 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:11.003298 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerDied","Data":"b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3"} Jul 30 08:19:11.193527 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:11.193488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-registry-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:11.193527 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:11.193540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:11.193980 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:11.193715 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:19:11.193980 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:11.193789 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls podName:efe7bb09-143e-4cc6-b7ef-b2446ff5fb35 nodeName:}" failed. No retries permitted until 2026-07-30 08:19:12.193768873 +0000 UTC m=+446.453715206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls") pod "feast-simple-feast-setup-554b65bbfb-l8wfh" (UID: "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35") : secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:19:11.195858 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:11.195838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-registry-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:12.202897 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:12.202861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:12.205211 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:12.205189 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls\") pod \"feast-simple-feast-setup-554b65bbfb-l8wfh\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:12.261012 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:12.260975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:12.404558 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:12.404514 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh"] Jul 30 08:19:12.406047 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:19:12.406019 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefe7bb09_143e_4cc6_b7ef_b2446ff5fb35.slice/crio-52ef558f1b9d674587ade3ef2860d726f4ee150f1dd0326959190ad33a142de2 WatchSource:0}: Error finding container 52ef558f1b9d674587ade3ef2860d726f4ee150f1dd0326959190ad33a142de2: Status 404 returned error can't find the container with id 52ef558f1b9d674587ade3ef2860d726f4ee150f1dd0326959190ad33a142de2 Jul 30 08:19:13.013725 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:13.013687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerStarted","Data":"3b507048aea1f25d097795b17b159a15df07bf912c7987cafd7b300546663d89"} Jul 30 08:19:13.013725 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:13.013731 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerStarted","Data":"52ef558f1b9d674587ade3ef2860d726f4ee150f1dd0326959190ad33a142de2"} Jul 30 08:19:17.746330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:17.746281 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.19:8443: connect: connection refused" Jul 30 08:19:17.770464 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:17.770422 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.19:6567/health\": dial tcp 10.134.0.19:6567: connect: connection refused" Jul 30 08:19:18.034292 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:18.034212 2576 generic.go:356] "Generic (PLEG): container finished" podID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerID="3b507048aea1f25d097795b17b159a15df07bf912c7987cafd7b300546663d89" exitCode=0 Jul 30 08:19:18.034449 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:18.034288 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerDied","Data":"3b507048aea1f25d097795b17b159a15df07bf912c7987cafd7b300546663d89"} Jul 30 08:19:19.040171 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:19.040133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerStarted","Data":"e12775ed1b72c1c55ffe74d65da3f0726470c7a273743885aa77a5cea60c35c7"} Jul 30 08:19:24.065473 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:24.065439 2576 generic.go:356] "Generic (PLEG): container finished" podID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerID="e12775ed1b72c1c55ffe74d65da3f0726470c7a273743885aa77a5cea60c35c7" exitCode=0 Jul 30 08:19:24.065814 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:24.065501 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerDied","Data":"e12775ed1b72c1c55ffe74d65da3f0726470c7a273743885aa77a5cea60c35c7"} Jul 30 08:19:25.075735 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:25.075693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerStarted","Data":"fdf4afc4cb3ba51dc43f4aeb17e271484b373846413143d2f68355deeca54351"} Jul 30 08:19:25.076208 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:25.075752 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerStarted","Data":"ac8d6bc13ba5b5da189ee5fe3c448214a4f1974d2d0f8f5e8dfabbcc49484662"} Jul 30 08:19:25.076208 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:25.075771 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerStarted","Data":"d94d3cc7629319e27812a6c83bd77debbaf0718b60e58df75119488961f2ee90"} Jul 30 08:19:25.076208 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:25.075786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerStarted","Data":"7cf89a1d921161336780ed9b0bc6f41dc9401b873d8399493b87a1b8ceca8c42"} Jul 30 08:19:25.104782 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:25.104716 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podStartSLOduration=15.104693393 podStartE2EDuration="15.104693393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:19:25.102955366 +0000 UTC m=+459.362901739" watchObservedRunningTime="2026-07-30 08:19:25.104693393 +0000 UTC m=+459.364639729" Jul 30 08:19:27.261868 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:27.261823 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:27.262353 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:27.261892 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:27.262353 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:27.261910 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:27.262353 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:27.262336 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:27.264395 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:27.264348 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.20:8816: connect: connection refused" Jul 30 08:19:27.264679 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:27.264447 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Jul 30 08:19:27.264778 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:27.264465 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Jul 30 08:19:27.264778 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:27.264500 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.20:6571: connect: connection refused" Jul 30 08:19:27.746216 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:27.746169 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.19:8443: connect: connection refused" Jul 30 08:19:27.771553 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:27.771506 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.19:6567/health\": dial tcp 10.134.0.19:6567: connect: connection refused" Jul 30 08:19:30.261627 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:30.261551 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Jul 30 08:19:30.262215 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:30.262184 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:30.262482 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:30.262462 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:30.262726 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:30.262703 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:30.262996 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:30.262973 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:30.267390 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:30.267343 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:31.105792 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:31.105763 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:31.105981 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:31.105811 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:31.108940 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:31.108911 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:32.109536 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:32.109508 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:32.261710 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:32.261673 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:33.262534 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:33.262503 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:34.117091 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:34.117064 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:19:37.746616 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:37.746577 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.19:8443: connect: connection refused" Jul 30 08:19:37.747101 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:37.746709 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:19:37.770531 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:37.770490 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.19:6567/health\": dial tcp 10.134.0.19:6567: connect: connection refused" Jul 30 08:19:37.770708 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:37.770687 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:19:40.803339 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.803307 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:19:40.862832 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.862799 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59444af2-8a57-4e5c-9bdd-8c727baa576e-feast-data\") pod \"59444af2-8a57-4e5c-9bdd-8c727baa576e\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " Jul 30 08:19:40.862988 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.862855 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-offline-tls\") pod \"59444af2-8a57-4e5c-9bdd-8c727baa576e\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " Jul 30 08:19:40.862988 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.862879 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-online-tls\") pod \"59444af2-8a57-4e5c-9bdd-8c727baa576e\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " Jul 30 08:19:40.862988 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.862901 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-ui-tls\") pod \"59444af2-8a57-4e5c-9bdd-8c727baa576e\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " Jul 30 08:19:40.862988 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.862942 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-registry-tls\") pod \"59444af2-8a57-4e5c-9bdd-8c727baa576e\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " Jul 30 08:19:40.862988 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.862969 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v758\" (UniqueName: \"kubernetes.io/projected/59444af2-8a57-4e5c-9bdd-8c727baa576e-kube-api-access-9v758\") pod \"59444af2-8a57-4e5c-9bdd-8c727baa576e\" (UID: \"59444af2-8a57-4e5c-9bdd-8c727baa576e\") " Jul 30 08:19:40.863432 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.863403 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59444af2-8a57-4e5c-9bdd-8c727baa576e-feast-data" (OuterVolumeSpecName: "feast-data") pod "59444af2-8a57-4e5c-9bdd-8c727baa576e" (UID: "59444af2-8a57-4e5c-9bdd-8c727baa576e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:19:40.865226 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.865199 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "59444af2-8a57-4e5c-9bdd-8c727baa576e" (UID: "59444af2-8a57-4e5c-9bdd-8c727baa576e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:19:40.865586 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.865554 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-online-tls" (OuterVolumeSpecName: "online-tls") pod "59444af2-8a57-4e5c-9bdd-8c727baa576e" (UID: "59444af2-8a57-4e5c-9bdd-8c727baa576e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:19:40.865586 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.865564 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59444af2-8a57-4e5c-9bdd-8c727baa576e-kube-api-access-9v758" (OuterVolumeSpecName: "kube-api-access-9v758") pod "59444af2-8a57-4e5c-9bdd-8c727baa576e" (UID: "59444af2-8a57-4e5c-9bdd-8c727baa576e"). InnerVolumeSpecName "kube-api-access-9v758". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:19:40.865728 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.865573 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "59444af2-8a57-4e5c-9bdd-8c727baa576e" (UID: "59444af2-8a57-4e5c-9bdd-8c727baa576e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:19:40.865728 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.865657 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "59444af2-8a57-4e5c-9bdd-8c727baa576e" (UID: "59444af2-8a57-4e5c-9bdd-8c727baa576e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:19:40.963855 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.963778 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-offline-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:19:40.963855 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.963829 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:19:40.963855 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.963841 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-ui-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:19:40.963855 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.963848 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59444af2-8a57-4e5c-9bdd-8c727baa576e-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:19:40.963855 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.963856 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v758\" (UniqueName: \"kubernetes.io/projected/59444af2-8a57-4e5c-9bdd-8c727baa576e-kube-api-access-9v758\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:19:40.964115 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:40.963865 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59444af2-8a57-4e5c-9bdd-8c727baa576e-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:19:41.148046 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.148006 2576 generic.go:356] "Generic (PLEG): container finished" podID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerID="a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55" exitCode=137 Jul 30 08:19:41.148046 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.148038 2576 generic.go:356] "Generic (PLEG): container finished" podID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerID="7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e" exitCode=137 Jul 30 08:19:41.148424 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.148048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerDied","Data":"a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55"} Jul 30 08:19:41.148424 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.148076 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" Jul 30 08:19:41.148424 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.148097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerDied","Data":"7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e"} Jul 30 08:19:41.148424 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.148114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv" event={"ID":"59444af2-8a57-4e5c-9bdd-8c727baa576e","Type":"ContainerDied","Data":"912eaa4f0cab7b1dd32590a6a4008ef495f761975c3a4a24ef7138a771010861"} Jul 30 08:19:41.148424 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.148125 2576 scope.go:117] "RemoveContainer" containerID="a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d" Jul 30 08:19:41.156655 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.156639 2576 scope.go:117] "RemoveContainer" containerID="a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55" Jul 30 08:19:41.164178 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.163993 2576 scope.go:117] "RemoveContainer" containerID="b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3" Jul 30 08:19:41.174571 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.174549 2576 scope.go:117] "RemoveContainer" containerID="7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e" Jul 30 08:19:41.183856 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.183836 2576 scope.go:117] "RemoveContainer" containerID="4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48" Jul 30 08:19:41.191962 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.191943 2576 scope.go:117] "RemoveContainer" containerID="084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a" Jul 30 08:19:41.192468 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.192444 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv"] Jul 30 08:19:41.199293 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.199273 2576 scope.go:117] "RemoveContainer" containerID="a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d" Jul 30 08:19:41.199575 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:41.199553 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d\": container with ID starting with a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d not found: ID does not exist" containerID="a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d" Jul 30 08:19:41.199639 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.199582 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d"} err="failed to get container status \"a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d\": rpc error: code = NotFound desc = could not find container \"a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d\": container with ID starting with a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d not found: ID does not exist" Jul 30 08:19:41.199639 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.199603 2576 scope.go:117] "RemoveContainer" containerID="a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55" Jul 30 08:19:41.199871 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:41.199855 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55\": container with ID starting with a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55 not found: ID does not exist" containerID="a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55" Jul 30 08:19:41.199921 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.199877 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55"} err="failed to get container status \"a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55\": rpc error: code = NotFound desc = could not find container \"a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55\": container with ID starting with a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55 not found: ID does not exist" Jul 30 08:19:41.199921 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.199900 2576 scope.go:117] "RemoveContainer" containerID="b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3" Jul 30 08:19:41.200137 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:41.200120 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3\": container with ID starting with b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3 not found: ID does not exist" containerID="b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3" Jul 30 08:19:41.200176 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.200142 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3"} err="failed to get container status \"b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3\": rpc error: code = NotFound desc = could not find container \"b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3\": container with ID starting with b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3 not found: ID does not exist" Jul 30 08:19:41.200176 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.200157 2576 scope.go:117] "RemoveContainer" containerID="7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e" Jul 30 08:19:41.200434 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:41.200412 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e\": container with ID starting with 7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e not found: ID does not exist" containerID="7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e" Jul 30 08:19:41.200484 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.200440 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e"} err="failed to get container status \"7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e\": rpc error: code = NotFound desc = could not find container \"7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e\": container with ID starting with 7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e not found: ID does not exist" Jul 30 08:19:41.200484 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.200458 2576 scope.go:117] "RemoveContainer" containerID="4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48" Jul 30 08:19:41.200698 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:41.200680 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48\": container with ID starting with 4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48 not found: ID does not exist" containerID="4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48" Jul 30 08:19:41.200742 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.200702 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48"} err="failed to get container status \"4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48\": rpc error: code = NotFound desc = could not find container \"4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48\": container with ID starting with 4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48 not found: ID does not exist" Jul 30 08:19:41.200742 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.200716 2576 scope.go:117] "RemoveContainer" containerID="084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a" Jul 30 08:19:41.200979 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:19:41.200955 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a\": container with ID starting with 084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a not found: ID does not exist" containerID="084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a" Jul 30 08:19:41.201078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.200985 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a"} err="failed to get container status \"084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a\": rpc error: code = NotFound desc = could not find container \"084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a\": container with ID starting with 084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a not found: ID does not exist" Jul 30 08:19:41.201078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.201027 2576 scope.go:117] "RemoveContainer" containerID="a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d" Jul 30 08:19:41.201341 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.201318 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d"} err="failed to get container status \"a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d\": rpc error: code = NotFound desc = could not find container \"a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d\": container with ID starting with a7aeb1e6f3a62061ac7ce2860457868ed968a6d11226174111a726e883df0d8d not found: ID does not exist" Jul 30 08:19:41.201406 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.201345 2576 scope.go:117] "RemoveContainer" containerID="a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55" Jul 30 08:19:41.201592 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.201563 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55"} err="failed to get container status \"a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55\": rpc error: code = NotFound desc = could not find container \"a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55\": container with ID starting with a3b37a9a343082e1fa7e2c1648756cd21f9cba7e2ed9364502b5dc13d1495a55 not found: ID does not exist" Jul 30 08:19:41.201746 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.201589 2576 scope.go:117] "RemoveContainer" containerID="b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3" Jul 30 08:19:41.201941 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.201909 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3"} err="failed to get container status \"b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3\": rpc error: code = NotFound desc = could not find container \"b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3\": container with ID starting with b3108f9903fca0f3f5fd0f3ad18bdef58225962cac1430fad3be807e0170acf3 not found: ID does not exist" Jul 30 08:19:41.202039 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.201943 2576 scope.go:117] "RemoveContainer" containerID="7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e" Jul 30 08:19:41.202366 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.202312 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e"} err="failed to get container status \"7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e\": rpc error: code = NotFound desc = could not find container \"7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e\": container with ID starting with 7cf3adb17cc472d0c14acf539d74d1eda9674697e047e5c47c06193fe887014e not found: ID does not exist" Jul 30 08:19:41.202366 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.202339 2576 scope.go:117] "RemoveContainer" containerID="4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48" Jul 30 08:19:41.202687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.202664 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48"} err="failed to get container status \"4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48\": rpc error: code = NotFound desc = could not find container \"4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48\": container with ID starting with 4cbcdb0c89315bf20bfdd325477a90b3797920d3768d13f00122e17e9a5b2b48 not found: ID does not exist" Jul 30 08:19:41.202763 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.202688 2576 scope.go:117] "RemoveContainer" containerID="084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a" Jul 30 08:19:41.202934 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.202915 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a"} err="failed to get container status \"084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a\": rpc error: code = NotFound desc = could not find container \"084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a\": container with ID starting with 084cf0e6b626e4ff1c8837fc43328718e5a6bf2daaad1e868cd2c70c2ea35b0a not found: ID does not exist" Jul 30 08:19:41.203519 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:41.203501 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-9szhv"] Jul 30 08:19:42.290713 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:19:42.290681 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" path="/var/lib/kubelet/pods/59444af2-8a57-4e5c-9bdd-8c727baa576e/volumes" Jul 30 08:20:36.174590 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174558 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn"] Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174857 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="offline" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174868 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="offline" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174877 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="ui" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174882 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="ui" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174892 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="online" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174897 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="online" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174907 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="feast-apply" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174912 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="feast-apply" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174919 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="feast-init" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174925 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="feast-init" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174931 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="registry" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174936 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="registry" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.174999 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="registry" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.175009 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="offline" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.175016 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="online" Jul 30 08:20:36.175108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.175022 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="59444af2-8a57-4e5c-9bdd-8c727baa576e" containerName="ui" Jul 30 08:20:36.178356 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.178339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.180860 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.180841 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 30 08:20:36.180972 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.180914 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 30 08:20:36.181037 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.180970 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 30 08:20:36.181037 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.180984 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 30 08:20:36.181200 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.181178 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 30 08:20:36.181310 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.181182 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 30 08:20:36.181310 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.181207 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 30 08:20:36.181310 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.181284 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-8dmqs\"" Jul 30 08:20:36.192465 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.192440 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn"] Jul 30 08:20:36.204926 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.204901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-online-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.205049 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.204933 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-offline-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.205049 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.204968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-feast-data\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.205049 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.205021 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-registry-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.205150 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.205066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-ui-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.205150 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.205101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4cth\" (UniqueName: \"kubernetes.io/projected/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-kube-api-access-g4cth\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.306348 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.306316 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-registry-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.306348 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.306350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-ui-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.306613 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.306388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cth\" (UniqueName: \"kubernetes.io/projected/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-kube-api-access-g4cth\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.306613 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.306423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-online-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.306613 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.306440 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-offline-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.306613 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.306465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-feast-data\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.306613 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:20:36.306543 2576 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jul 30 08:20:36.306834 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:20:36.306641 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-online-tls podName:117162f0-f7d7-45c8-afd6-d39d8e5b6aa5 nodeName:}" failed. No retries permitted until 2026-07-30 08:20:36.806617689 +0000 UTC m=+531.066564007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-online-tls") pod "feast-simple-feast-remote-setup-6c476c499c-2hhkn" (UID: "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5") : secret "feast-simple-feast-remote-setup-online-tls" not found Jul 30 08:20:36.306897 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.306851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-feast-data\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.307084 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.307064 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-registry-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.308855 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.308837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-ui-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.308974 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.308915 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-offline-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.318032 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.317999 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cth\" (UniqueName: \"kubernetes.io/projected/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-kube-api-access-g4cth\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.810988 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.810940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-online-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:36.813230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:36.813211 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-online-tls\") pod \"feast-simple-feast-remote-setup-6c476c499c-2hhkn\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:37.087627 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:37.087528 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:37.276079 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:20:37.276046 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod117162f0_f7d7_45c8_afd6_d39d8e5b6aa5.slice/crio-c42b33a9faf8cdf04dbe3685391c7b14f8d622a6b5ef393a06d2e4be1f513957 WatchSource:0}: Error finding container c42b33a9faf8cdf04dbe3685391c7b14f8d622a6b5ef393a06d2e4be1f513957: Status 404 returned error can't find the container with id c42b33a9faf8cdf04dbe3685391c7b14f8d622a6b5ef393a06d2e4be1f513957 Jul 30 08:20:37.276880 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:37.276857 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn"] Jul 30 08:20:37.362015 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:37.361915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerStarted","Data":"f79743c0bfb902563f780e65a6f70da20e3b6ab1e06114ae1c36b05c1f02e381"} Jul 30 08:20:37.362015 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:37.361968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerStarted","Data":"c42b33a9faf8cdf04dbe3685391c7b14f8d622a6b5ef393a06d2e4be1f513957"} Jul 30 08:20:42.381968 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:42.381934 2576 generic.go:356] "Generic (PLEG): container finished" podID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerID="f79743c0bfb902563f780e65a6f70da20e3b6ab1e06114ae1c36b05c1f02e381" exitCode=0 Jul 30 08:20:42.381968 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:42.381974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerDied","Data":"f79743c0bfb902563f780e65a6f70da20e3b6ab1e06114ae1c36b05c1f02e381"} Jul 30 08:20:43.387869 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:43.387833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerStarted","Data":"336f6473258e5bfd3ff1ae57803104c858ea198c45d3bdc722617f6487e8abfa"} Jul 30 08:20:48.408077 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:48.408044 2576 generic.go:356] "Generic (PLEG): container finished" podID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerID="336f6473258e5bfd3ff1ae57803104c858ea198c45d3bdc722617f6487e8abfa" exitCode=0 Jul 30 08:20:48.408449 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:48.408119 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerDied","Data":"336f6473258e5bfd3ff1ae57803104c858ea198c45d3bdc722617f6487e8abfa"} Jul 30 08:20:48.412649 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:48.412628 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 30 08:20:49.417000 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:49.416953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerStarted","Data":"3d8a7c2871d12add6281ff9bb4967a78381ef483fc10e562ea84ed1a61226f91"} Jul 30 08:20:49.417416 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:49.417022 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerStarted","Data":"e7fe8fe4a7ac6dcc1ae0aae18bf58c0044aa1f3e1191aa7918aa256ce81a6714"} Jul 30 08:20:49.417416 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:49.417046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerStarted","Data":"59e9a142c60b996f123ebf4f8bfe8f66d10636d81dac590bc92b9fef2ccd4a2c"} Jul 30 08:20:49.445396 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:49.443503 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podStartSLOduration=13.443471998 podStartE2EDuration="13.443471998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:20:49.440273846 +0000 UTC m=+543.700220219" watchObservedRunningTime="2026-07-30 08:20:49.443471998 +0000 UTC m=+543.703418334" Jul 30 08:20:52.088479 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:52.088428 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:52.088963 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:52.088510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:52.088963 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:52.088531 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:52.090179 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:52.090145 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Jul 30 08:20:52.090356 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:52.090178 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Jul 30 08:20:52.090356 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:52.090154 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.21:8816: connect: connection refused" Jul 30 08:20:55.089061 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:55.089029 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:55.089549 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:55.089212 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:55.089617 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:55.089586 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:55.089675 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:55.089625 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:55.089806 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:55.089784 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:55.090101 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:55.090082 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:55.095490 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:55.095469 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:55.444691 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:55.444604 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:20:55.449034 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:20:55.449008 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:21:46.146522 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:46.146489 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:21:46.151744 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:46.151722 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:21:46.152658 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:46.152639 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:21:46.158144 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:46.158127 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:21:57.607470 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:57.607427 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn"] Jul 30 08:21:57.609762 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:57.607670 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="online" containerID="cri-o://59e9a142c60b996f123ebf4f8bfe8f66d10636d81dac590bc92b9fef2ccd4a2c" gracePeriod=30 Jul 30 08:21:57.609762 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:57.607698 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="ui" containerID="cri-o://3d8a7c2871d12add6281ff9bb4967a78381ef483fc10e562ea84ed1a61226f91" gracePeriod=30 Jul 30 08:21:57.609762 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:57.607741 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="offline" containerID="cri-o://e7fe8fe4a7ac6dcc1ae0aae18bf58c0044aa1f3e1191aa7918aa256ce81a6714" gracePeriod=30 Jul 30 08:21:57.719436 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:57.719405 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh"] Jul 30 08:21:57.719833 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:57.719661 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="offline" containerID="cri-o://ac8d6bc13ba5b5da189ee5fe3c448214a4f1974d2d0f8f5e8dfabbcc49484662" gracePeriod=30 Jul 30 08:21:57.719833 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:57.719685 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="registry" containerID="cri-o://7cf89a1d921161336780ed9b0bc6f41dc9401b873d8399493b87a1b8ceca8c42" gracePeriod=30 Jul 30 08:21:57.719833 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:57.719735 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="online" containerID="cri-o://d94d3cc7629319e27812a6c83bd77debbaf0718b60e58df75119488961f2ee90" gracePeriod=30 Jul 30 08:21:57.719833 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:57.719661 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="ui" containerID="cri-o://fdf4afc4cb3ba51dc43f4aeb17e271484b373846413143d2f68355deeca54351" gracePeriod=30 Jul 30 08:21:58.684560 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:58.684524 2576 generic.go:356] "Generic (PLEG): container finished" podID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerID="fdf4afc4cb3ba51dc43f4aeb17e271484b373846413143d2f68355deeca54351" exitCode=0 Jul 30 08:21:58.684560 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:58.684550 2576 generic.go:356] "Generic (PLEG): container finished" podID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerID="d94d3cc7629319e27812a6c83bd77debbaf0718b60e58df75119488961f2ee90" exitCode=0 Jul 30 08:21:58.685026 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:58.684599 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerDied","Data":"fdf4afc4cb3ba51dc43f4aeb17e271484b373846413143d2f68355deeca54351"} Jul 30 08:21:58.685026 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:58.684654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerDied","Data":"d94d3cc7629319e27812a6c83bd77debbaf0718b60e58df75119488961f2ee90"} Jul 30 08:21:58.687050 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:58.687027 2576 generic.go:356] "Generic (PLEG): container finished" podID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerID="3d8a7c2871d12add6281ff9bb4967a78381ef483fc10e562ea84ed1a61226f91" exitCode=0 Jul 30 08:21:58.687050 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:58.687047 2576 generic.go:356] "Generic (PLEG): container finished" podID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerID="59e9a142c60b996f123ebf4f8bfe8f66d10636d81dac590bc92b9fef2ccd4a2c" exitCode=0 Jul 30 08:21:58.687181 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:58.687086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerDied","Data":"3d8a7c2871d12add6281ff9bb4967a78381ef483fc10e562ea84ed1a61226f91"} Jul 30 08:21:58.687181 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:21:58.687112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerDied","Data":"59e9a142c60b996f123ebf4f8bfe8f66d10636d81dac590bc92b9fef2ccd4a2c"} Jul 30 08:22:01.106687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:01.106643 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Jul 30 08:22:04.117442 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:04.117402 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Jul 30 08:22:05.089792 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:05.089749 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Jul 30 08:22:05.445064 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:05.444960 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Jul 30 08:22:11.106902 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:11.106806 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Jul 30 08:22:14.116774 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:14.116721 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Jul 30 08:22:15.090443 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:15.090402 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Jul 30 08:22:15.445196 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:15.445091 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Jul 30 08:22:21.106025 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:21.105976 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Jul 30 08:22:21.106502 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:21.106089 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:22:24.117531 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:24.117487 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Jul 30 08:22:24.117914 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:24.117688 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:22:25.089836 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:25.089793 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Jul 30 08:22:25.090014 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:25.089918 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:22:25.444940 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:25.444845 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Jul 30 08:22:25.445306 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:25.444952 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:22:27.777610 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:22:27.777565 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefe7bb09_143e_4cc6_b7ef_b2446ff5fb35.slice/crio-conmon-7cf89a1d921161336780ed9b0bc6f41dc9401b873d8399493b87a1b8ceca8c42.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefe7bb09_143e_4cc6_b7ef_b2446ff5fb35.slice/crio-conmon-ac8d6bc13ba5b5da189ee5fe3c448214a4f1974d2d0f8f5e8dfabbcc49484662.scope\": RecentStats: unable to find data in memory cache]" Jul 30 08:22:27.803441 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:27.803404 2576 generic.go:356] "Generic (PLEG): container finished" podID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerID="ac8d6bc13ba5b5da189ee5fe3c448214a4f1974d2d0f8f5e8dfabbcc49484662" exitCode=137 Jul 30 08:22:27.803441 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:27.803429 2576 generic.go:356] "Generic (PLEG): container finished" podID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerID="7cf89a1d921161336780ed9b0bc6f41dc9401b873d8399493b87a1b8ceca8c42" exitCode=137 Jul 30 08:22:27.803647 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:27.803460 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerDied","Data":"ac8d6bc13ba5b5da189ee5fe3c448214a4f1974d2d0f8f5e8dfabbcc49484662"} Jul 30 08:22:27.803647 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:27.803501 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerDied","Data":"7cf89a1d921161336780ed9b0bc6f41dc9401b873d8399493b87a1b8ceca8c42"} Jul 30 08:22:27.806001 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:27.805978 2576 generic.go:356] "Generic (PLEG): container finished" podID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerID="e7fe8fe4a7ac6dcc1ae0aae18bf58c0044aa1f3e1191aa7918aa256ce81a6714" exitCode=137 Jul 30 08:22:27.806113 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:27.806010 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerDied","Data":"e7fe8fe4a7ac6dcc1ae0aae18bf58c0044aa1f3e1191aa7918aa256ce81a6714"} Jul 30 08:22:28.269296 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.269265 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:22:28.366011 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.365984 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:22:28.431780 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.431683 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-registry-tls\") pod \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " Jul 30 08:22:28.431780 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.431718 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-online-tls\") pod \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " Jul 30 08:22:28.431780 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.431735 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-registry-tls\") pod \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " Jul 30 08:22:28.431780 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.431773 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-feast-data\") pod \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " Jul 30 08:22:28.432141 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.431798 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-offline-tls\") pod \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " Jul 30 08:22:28.432141 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.431818 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-online-tls\") pod \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " Jul 30 08:22:28.432141 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.431850 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7ptl\" (UniqueName: \"kubernetes.io/projected/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-kube-api-access-b7ptl\") pod \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " Jul 30 08:22:28.432141 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.431874 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-ui-tls\") pod \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " Jul 30 08:22:28.432141 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.431900 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4cth\" (UniqueName: \"kubernetes.io/projected/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-kube-api-access-g4cth\") pod \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " Jul 30 08:22:28.432141 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.431941 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-ui-tls\") pod \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\" (UID: \"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5\") " Jul 30 08:22:28.432141 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.431975 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-feast-data\") pod \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " Jul 30 08:22:28.432141 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.432000 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls\") pod \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\" (UID: \"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35\") " Jul 30 08:22:28.432141 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.432110 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" (UID: "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 08:22:28.432615 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.432247 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.432615 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.432314 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-feast-data" (OuterVolumeSpecName: "feast-data") pod "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" (UID: "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:22:28.433029 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.432999 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-feast-data" (OuterVolumeSpecName: "feast-data") pod "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" (UID: "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:22:28.434791 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.434744 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" (UID: "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:22:28.434900 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.434824 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" (UID: "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:22:28.434900 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.434836 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-kube-api-access-b7ptl" (OuterVolumeSpecName: "kube-api-access-b7ptl") pod "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" (UID: "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35"). InnerVolumeSpecName "kube-api-access-b7ptl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:22:28.435024 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.434943 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" (UID: "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:22:28.435024 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.434954 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-kube-api-access-g4cth" (OuterVolumeSpecName: "kube-api-access-g4cth") pod "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" (UID: "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5"). InnerVolumeSpecName "kube-api-access-g4cth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:22:28.435105 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.435067 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-online-tls" (OuterVolumeSpecName: "online-tls") pod "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" (UID: "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:22:28.435105 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.435090 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" (UID: "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:22:28.435313 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.435295 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-online-tls" (OuterVolumeSpecName: "online-tls") pod "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" (UID: "117162f0-f7d7-45c8-afd6-d39d8e5b6aa5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:22:28.435896 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.435875 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" (UID: "efe7bb09-143e-4cc6-b7ef-b2446ff5fb35"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:22:28.532848 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.532810 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.532848 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.532842 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.532848 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.532854 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.533094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.532864 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-offline-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.533094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.532876 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.533094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.532887 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ptl\" (UniqueName: \"kubernetes.io/projected/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-kube-api-access-b7ptl\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.533094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.532901 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-ui-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.533094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.532914 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4cth\" (UniqueName: \"kubernetes.io/projected/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-kube-api-access-g4cth\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.533094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.532926 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5-ui-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.533094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.532939 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.533094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.532950 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35-offline-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:22:28.813187 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.813158 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" Jul 30 08:22:28.813187 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.813177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh" event={"ID":"efe7bb09-143e-4cc6-b7ef-b2446ff5fb35","Type":"ContainerDied","Data":"52ef558f1b9d674587ade3ef2860d726f4ee150f1dd0326959190ad33a142de2"} Jul 30 08:22:28.813682 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.813232 2576 scope.go:117] "RemoveContainer" containerID="fdf4afc4cb3ba51dc43f4aeb17e271484b373846413143d2f68355deeca54351" Jul 30 08:22:28.816118 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.816092 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" event={"ID":"117162f0-f7d7-45c8-afd6-d39d8e5b6aa5","Type":"ContainerDied","Data":"c42b33a9faf8cdf04dbe3685391c7b14f8d622a6b5ef393a06d2e4be1f513957"} Jul 30 08:22:28.816233 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.816169 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn" Jul 30 08:22:28.822305 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.822262 2576 scope.go:117] "RemoveContainer" containerID="ac8d6bc13ba5b5da189ee5fe3c448214a4f1974d2d0f8f5e8dfabbcc49484662" Jul 30 08:22:28.838515 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.838139 2576 scope.go:117] "RemoveContainer" containerID="d94d3cc7629319e27812a6c83bd77debbaf0718b60e58df75119488961f2ee90" Jul 30 08:22:28.842544 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.842517 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh"] Jul 30 08:22:28.846395 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.846335 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-554b65bbfb-l8wfh"] Jul 30 08:22:28.847699 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.847682 2576 scope.go:117] "RemoveContainer" containerID="7cf89a1d921161336780ed9b0bc6f41dc9401b873d8399493b87a1b8ceca8c42" Jul 30 08:22:28.855192 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.855174 2576 scope.go:117] "RemoveContainer" containerID="e12775ed1b72c1c55ffe74d65da3f0726470c7a273743885aa77a5cea60c35c7" Jul 30 08:22:28.857932 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.857910 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn"] Jul 30 08:22:28.861476 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.861449 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6c476c499c-2hhkn"] Jul 30 08:22:28.863627 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.863611 2576 scope.go:117] "RemoveContainer" containerID="3b507048aea1f25d097795b17b159a15df07bf912c7987cafd7b300546663d89" Jul 30 08:22:28.870831 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.870811 2576 scope.go:117] "RemoveContainer" containerID="3d8a7c2871d12add6281ff9bb4967a78381ef483fc10e562ea84ed1a61226f91" Jul 30 08:22:28.878212 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.878189 2576 scope.go:117] "RemoveContainer" containerID="e7fe8fe4a7ac6dcc1ae0aae18bf58c0044aa1f3e1191aa7918aa256ce81a6714" Jul 30 08:22:28.885732 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.885715 2576 scope.go:117] "RemoveContainer" containerID="59e9a142c60b996f123ebf4f8bfe8f66d10636d81dac590bc92b9fef2ccd4a2c" Jul 30 08:22:28.893033 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.893014 2576 scope.go:117] "RemoveContainer" containerID="336f6473258e5bfd3ff1ae57803104c858ea198c45d3bdc722617f6487e8abfa" Jul 30 08:22:28.900231 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:28.900212 2576 scope.go:117] "RemoveContainer" containerID="f79743c0bfb902563f780e65a6f70da20e3b6ab1e06114ae1c36b05c1f02e381" Jul 30 08:22:30.291311 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:30.291276 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" path="/var/lib/kubelet/pods/117162f0-f7d7-45c8-afd6-d39d8e5b6aa5/volumes" Jul 30 08:22:30.291816 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:30.291802 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" path="/var/lib/kubelet/pods/efe7bb09-143e-4cc6-b7ef-b2446ff5fb35/volumes" Jul 30 08:22:40.599451 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599414 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-x6ts5"] Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599688 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="online" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599700 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="online" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599709 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="ui" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599714 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="ui" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599721 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="feast-init" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599727 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="feast-init" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599733 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="online" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599738 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="online" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599744 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="registry" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599753 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="registry" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599762 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="feast-init" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599768 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="feast-init" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599774 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="ui" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599779 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="ui" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599791 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="feast-apply" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599797 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="feast-apply" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599803 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="offline" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599808 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="offline" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599814 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="offline" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599819 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="offline" Jul 30 08:22:40.599820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599825 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="feast-apply" Jul 30 08:22:40.600746 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599832 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="feast-apply" Jul 30 08:22:40.600746 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599883 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="registry" Jul 30 08:22:40.600746 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599894 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="online" Jul 30 08:22:40.600746 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599903 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="ui" Jul 30 08:22:40.600746 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599913 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="ui" Jul 30 08:22:40.600746 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599923 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="online" Jul 30 08:22:40.600746 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599934 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="117162f0-f7d7-45c8-afd6-d39d8e5b6aa5" containerName="offline" Jul 30 08:22:40.600746 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.599943 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe7bb09-143e-4cc6-b7ef-b2446ff5fb35" containerName="offline" Jul 30 08:22:40.602610 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.602592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-x6ts5" Jul 30 08:22:40.604953 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.604930 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 30 08:22:40.605075 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.604930 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jul 30 08:22:40.605788 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.605773 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 30 08:22:40.605884 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.605771 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-hmlrj\"" Jul 30 08:22:40.609388 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.609348 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-x6ts5"] Jul 30 08:22:40.626318 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.626286 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c73ce25c-f791-4966-b006-6a3cdc5302f1-postgresdata\") pod \"postgres-864846d4f5-x6ts5\" (UID: \"c73ce25c-f791-4966-b006-6a3cdc5302f1\") " pod="test-ns-feast/postgres-864846d4f5-x6ts5" Jul 30 08:22:40.626730 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.626704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd95t\" (UniqueName: \"kubernetes.io/projected/c73ce25c-f791-4966-b006-6a3cdc5302f1-kube-api-access-cd95t\") pod \"postgres-864846d4f5-x6ts5\" (UID: \"c73ce25c-f791-4966-b006-6a3cdc5302f1\") " pod="test-ns-feast/postgres-864846d4f5-x6ts5" Jul 30 08:22:40.727632 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.727596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c73ce25c-f791-4966-b006-6a3cdc5302f1-postgresdata\") pod \"postgres-864846d4f5-x6ts5\" (UID: \"c73ce25c-f791-4966-b006-6a3cdc5302f1\") " pod="test-ns-feast/postgres-864846d4f5-x6ts5" Jul 30 08:22:40.727632 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.727634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd95t\" (UniqueName: \"kubernetes.io/projected/c73ce25c-f791-4966-b006-6a3cdc5302f1-kube-api-access-cd95t\") pod \"postgres-864846d4f5-x6ts5\" (UID: \"c73ce25c-f791-4966-b006-6a3cdc5302f1\") " pod="test-ns-feast/postgres-864846d4f5-x6ts5" Jul 30 08:22:40.728016 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.727995 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c73ce25c-f791-4966-b006-6a3cdc5302f1-postgresdata\") pod \"postgres-864846d4f5-x6ts5\" (UID: \"c73ce25c-f791-4966-b006-6a3cdc5302f1\") " pod="test-ns-feast/postgres-864846d4f5-x6ts5" Jul 30 08:22:40.735428 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.735400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd95t\" (UniqueName: \"kubernetes.io/projected/c73ce25c-f791-4966-b006-6a3cdc5302f1-kube-api-access-cd95t\") pod \"postgres-864846d4f5-x6ts5\" (UID: \"c73ce25c-f791-4966-b006-6a3cdc5302f1\") " pod="test-ns-feast/postgres-864846d4f5-x6ts5" Jul 30 08:22:40.922515 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:40.922432 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-x6ts5" Jul 30 08:22:41.046078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:41.046047 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-x6ts5"] Jul 30 08:22:41.049402 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:22:41.049364 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc73ce25c_f791_4966_b006_6a3cdc5302f1.slice/crio-4b2f3938698f6b854853403b996710eb2d96aa09ae6af321d4b82a436f82db1d WatchSource:0}: Error finding container 4b2f3938698f6b854853403b996710eb2d96aa09ae6af321d4b82a436f82db1d: Status 404 returned error can't find the container with id 4b2f3938698f6b854853403b996710eb2d96aa09ae6af321d4b82a436f82db1d Jul 30 08:22:41.051160 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:41.051142 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 08:22:41.867769 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:41.867724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-x6ts5" event={"ID":"c73ce25c-f791-4966-b006-6a3cdc5302f1","Type":"ContainerStarted","Data":"4b2f3938698f6b854853403b996710eb2d96aa09ae6af321d4b82a436f82db1d"} Jul 30 08:22:45.886904 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:45.886870 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-x6ts5" event={"ID":"c73ce25c-f791-4966-b006-6a3cdc5302f1","Type":"ContainerStarted","Data":"0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf"} Jul 30 08:22:45.902861 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:45.902809 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-x6ts5" podStartSLOduration=1.157708415 podStartE2EDuration="5.902787393s" totalImagesPullingTime="4.745078978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:22:45.902490416 +0000 UTC m=+660.162436762" watchObservedRunningTime="2026-07-30 08:22:45.902787393 +0000 UTC m=+660.162733729" Jul 30 08:22:49.355826 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.355794 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq"] Jul 30 08:22:49.359148 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.359127 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.363325 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.363303 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jul 30 08:22:49.363463 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.363358 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jul 30 08:22:49.363538 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.363476 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jul 30 08:22:49.363850 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.363834 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-j989d\"" Jul 30 08:22:49.364048 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.364032 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jul 30 08:22:49.378283 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.378257 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq"] Jul 30 08:22:49.399384 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.399354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-offline-tls\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.399527 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.399420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt68k\" (UniqueName: \"kubernetes.io/projected/7c24562d-b637-41de-bc75-77df719719f6-kube-api-access-tt68k\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.399527 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.399485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-registry-tls\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.399652 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.399567 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-online-tls\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.399652 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.399612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7c24562d-b637-41de-bc75-77df719719f6-feast-data\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.500680 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.500641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-online-tls\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.500831 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.500692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7c24562d-b637-41de-bc75-77df719719f6-feast-data\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.500831 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.500737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-offline-tls\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.500831 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:22:49.500762 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Jul 30 08:22:49.500831 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.500778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt68k\" (UniqueName: \"kubernetes.io/projected/7c24562d-b637-41de-bc75-77df719719f6-kube-api-access-tt68k\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.500831 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.500811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-registry-tls\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.500831 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:22:49.500829 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-online-tls podName:7c24562d-b637-41de-bc75-77df719719f6 nodeName:}" failed. No retries permitted until 2026-07-30 08:22:50.000808397 +0000 UTC m=+664.260754726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-online-tls") pod "feast-credit-scoring-5969fbffbd-cz6rq" (UID: "7c24562d-b637-41de-bc75-77df719719f6") : secret "feast-credit-scoring-online-tls" not found Jul 30 08:22:49.501108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.501060 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7c24562d-b637-41de-bc75-77df719719f6-feast-data\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.503072 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.503051 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-registry-tls\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.503180 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.503094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-offline-tls\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:49.510691 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:49.510664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt68k\" (UniqueName: \"kubernetes.io/projected/7c24562d-b637-41de-bc75-77df719719f6-kube-api-access-tt68k\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:50.005694 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:50.005651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-online-tls\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:50.008042 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:50.008016 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-online-tls\") pod \"feast-credit-scoring-5969fbffbd-cz6rq\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:50.268306 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:50.268211 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:22:50.439247 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:50.439222 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq"] Jul 30 08:22:50.441234 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:22:50.441210 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c24562d_b637_41de_bc75_77df719719f6.slice/crio-3aed4a1d0797be073fb90f0f84feb1496bf31fe761cc17b8ed67a41d7e7e9232 WatchSource:0}: Error finding container 3aed4a1d0797be073fb90f0f84feb1496bf31fe761cc17b8ed67a41d7e7e9232: Status 404 returned error can't find the container with id 3aed4a1d0797be073fb90f0f84feb1496bf31fe761cc17b8ed67a41d7e7e9232 Jul 30 08:22:50.912802 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:50.912764 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerStarted","Data":"a80af46e0b5206b305da7b2e98010d54e4f71744c2fe60db0773759fddf547d9"} Jul 30 08:22:50.912993 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:50.912843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerStarted","Data":"3aed4a1d0797be073fb90f0f84feb1496bf31fe761cc17b8ed67a41d7e7e9232"} Jul 30 08:22:51.917736 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:51.917698 2576 generic.go:356] "Generic (PLEG): container finished" podID="7c24562d-b637-41de-bc75-77df719719f6" containerID="a80af46e0b5206b305da7b2e98010d54e4f71744c2fe60db0773759fddf547d9" exitCode=0 Jul 30 08:22:51.918092 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:51.917789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerDied","Data":"a80af46e0b5206b305da7b2e98010d54e4f71744c2fe60db0773759fddf547d9"} Jul 30 08:22:52.924125 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:52.924083 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerStarted","Data":"784bb3aae625f2f10e682315b20844bf51d87d13b6969520d123986f67d67d09"} Jul 30 08:22:58.950770 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:58.950737 2576 generic.go:356] "Generic (PLEG): container finished" podID="7c24562d-b637-41de-bc75-77df719719f6" containerID="784bb3aae625f2f10e682315b20844bf51d87d13b6969520d123986f67d67d09" exitCode=0 Jul 30 08:22:58.951343 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:58.950814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerDied","Data":"784bb3aae625f2f10e682315b20844bf51d87d13b6969520d123986f67d67d09"} Jul 30 08:22:59.959238 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:59.959202 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerStarted","Data":"a23a766f9fa9462111a9c9f2ca97b40ac327917844d0b0dce804dea076b86435"} Jul 30 08:22:59.959634 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:59.959275 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerStarted","Data":"1314be0b60020a686fa2361f3427069c25c29accd4728f05bc8e8942cc8e120b"} Jul 30 08:22:59.959634 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:59.959305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerStarted","Data":"4d6541de1c45d23f47065672439d2fd03997d79ad39fd57a1991ef85619f3958"} Jul 30 08:22:59.991064 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:22:59.990999 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podStartSLOduration=10.99097913 podStartE2EDuration="10.99097913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:22:59.989645415 +0000 UTC m=+674.249591777" watchObservedRunningTime="2026-07-30 08:22:59.99097913 +0000 UTC m=+674.250925464" Jul 30 08:23:00.269151 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:00.269040 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:23:00.269151 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:00.269116 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:23:00.269151 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:00.269136 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:23:02.269131 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:02.269080 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:23:02.269613 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:02.269210 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:23:02.269613 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:02.269239 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:23:02.271170 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:02.271134 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Jul 30 08:23:02.271170 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:02.271150 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Jul 30 08:23:02.271394 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:02.271149 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Jul 30 08:23:05.269202 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:05.268810 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Jul 30 08:23:05.269679 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:05.269643 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:23:05.270539 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:05.270517 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:23:05.274382 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:05.274347 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:23:05.988293 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:05.988262 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:23:08.269335 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:08.269295 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Jul 30 08:23:11.269708 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:11.269678 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:23:11.270477 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:23:11.270452 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:24:14.932336 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:14.932300 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq"] Jul 30 08:24:14.932922 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:14.932547 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="registry" containerID="cri-o://4d6541de1c45d23f47065672439d2fd03997d79ad39fd57a1991ef85619f3958" gracePeriod=30 Jul 30 08:24:14.932922 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:14.932621 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="offline" containerID="cri-o://a23a766f9fa9462111a9c9f2ca97b40ac327917844d0b0dce804dea076b86435" gracePeriod=30 Jul 30 08:24:14.932922 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:14.932613 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="online" containerID="cri-o://1314be0b60020a686fa2361f3427069c25c29accd4728f05bc8e8942cc8e120b" gracePeriod=30 Jul 30 08:24:14.949404 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:14.949362 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-x6ts5"] Jul 30 08:24:14.949575 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:14.949543 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-x6ts5" podUID="c73ce25c-f791-4966-b006-6a3cdc5302f1" containerName="postgres" containerID="cri-o://0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf" gracePeriod=30 Jul 30 08:24:15.189323 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.189261 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-x6ts5" Jul 30 08:24:15.248992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.248957 2576 generic.go:356] "Generic (PLEG): container finished" podID="c73ce25c-f791-4966-b006-6a3cdc5302f1" containerID="0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf" exitCode=0 Jul 30 08:24:15.248992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.248992 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-x6ts5" Jul 30 08:24:15.249180 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.249031 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-x6ts5" event={"ID":"c73ce25c-f791-4966-b006-6a3cdc5302f1","Type":"ContainerDied","Data":"0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf"} Jul 30 08:24:15.249180 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.249078 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-x6ts5" event={"ID":"c73ce25c-f791-4966-b006-6a3cdc5302f1","Type":"ContainerDied","Data":"4b2f3938698f6b854853403b996710eb2d96aa09ae6af321d4b82a436f82db1d"} Jul 30 08:24:15.249180 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.249100 2576 scope.go:117] "RemoveContainer" containerID="0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf" Jul 30 08:24:15.256961 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.256943 2576 scope.go:117] "RemoveContainer" containerID="0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf" Jul 30 08:24:15.257230 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:24:15.257213 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf\": container with ID starting with 0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf not found: ID does not exist" containerID="0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf" Jul 30 08:24:15.257282 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.257238 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf"} err="failed to get container status \"0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf\": rpc error: code = NotFound desc = could not find container \"0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf\": container with ID starting with 0f0c27a6bd42483320ee18249127abb11e8e8de1729e02e215d43d3f5653b6cf not found: ID does not exist" Jul 30 08:24:15.297659 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.297630 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c73ce25c-f791-4966-b006-6a3cdc5302f1-postgresdata\") pod \"c73ce25c-f791-4966-b006-6a3cdc5302f1\" (UID: \"c73ce25c-f791-4966-b006-6a3cdc5302f1\") " Jul 30 08:24:15.297827 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.297726 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd95t\" (UniqueName: \"kubernetes.io/projected/c73ce25c-f791-4966-b006-6a3cdc5302f1-kube-api-access-cd95t\") pod \"c73ce25c-f791-4966-b006-6a3cdc5302f1\" (UID: \"c73ce25c-f791-4966-b006-6a3cdc5302f1\") " Jul 30 08:24:15.299686 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.299660 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73ce25c-f791-4966-b006-6a3cdc5302f1-kube-api-access-cd95t" (OuterVolumeSpecName: "kube-api-access-cd95t") pod "c73ce25c-f791-4966-b006-6a3cdc5302f1" (UID: "c73ce25c-f791-4966-b006-6a3cdc5302f1"). InnerVolumeSpecName "kube-api-access-cd95t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:24:15.329222 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.329186 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73ce25c-f791-4966-b006-6a3cdc5302f1-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "c73ce25c-f791-4966-b006-6a3cdc5302f1" (UID: "c73ce25c-f791-4966-b006-6a3cdc5302f1"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:24:15.398608 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.398567 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd95t\" (UniqueName: \"kubernetes.io/projected/c73ce25c-f791-4966-b006-6a3cdc5302f1-kube-api-access-cd95t\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:24:15.398608 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.398618 2576 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c73ce25c-f791-4966-b006-6a3cdc5302f1-postgresdata\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:24:15.570693 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.570652 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-x6ts5"] Jul 30 08:24:15.575697 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.575671 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-x6ts5"] Jul 30 08:24:15.985780 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:15.985682 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Jul 30 08:24:16.255507 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:16.255422 2576 generic.go:356] "Generic (PLEG): container finished" podID="7c24562d-b637-41de-bc75-77df719719f6" containerID="1314be0b60020a686fa2361f3427069c25c29accd4728f05bc8e8942cc8e120b" exitCode=0 Jul 30 08:24:16.255507 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:16.255493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerDied","Data":"1314be0b60020a686fa2361f3427069c25c29accd4728f05bc8e8942cc8e120b"} Jul 30 08:24:16.290593 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:16.290558 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73ce25c-f791-4966-b006-6a3cdc5302f1" path="/var/lib/kubelet/pods/c73ce25c-f791-4966-b006-6a3cdc5302f1/volumes" Jul 30 08:24:25.985534 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:25.985492 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Jul 30 08:24:35.986200 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:35.986159 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Jul 30 08:24:35.986723 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:35.986282 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:24:45.271257 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.271168 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Jul 30 08:24:45.357901 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.357869 2576 generic.go:356] "Generic (PLEG): container finished" podID="7c24562d-b637-41de-bc75-77df719719f6" containerID="a23a766f9fa9462111a9c9f2ca97b40ac327917844d0b0dce804dea076b86435" exitCode=137 Jul 30 08:24:45.357901 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.357891 2576 generic.go:356] "Generic (PLEG): container finished" podID="7c24562d-b637-41de-bc75-77df719719f6" containerID="4d6541de1c45d23f47065672439d2fd03997d79ad39fd57a1991ef85619f3958" exitCode=137 Jul 30 08:24:45.358086 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.357941 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerDied","Data":"a23a766f9fa9462111a9c9f2ca97b40ac327917844d0b0dce804dea076b86435"} Jul 30 08:24:45.358086 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.358017 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerDied","Data":"4d6541de1c45d23f47065672439d2fd03997d79ad39fd57a1991ef85619f3958"} Jul 30 08:24:45.578474 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.578451 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:24:45.615791 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.615761 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-registry-tls\") pod \"7c24562d-b637-41de-bc75-77df719719f6\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " Jul 30 08:24:45.616025 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.615813 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt68k\" (UniqueName: \"kubernetes.io/projected/7c24562d-b637-41de-bc75-77df719719f6-kube-api-access-tt68k\") pod \"7c24562d-b637-41de-bc75-77df719719f6\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " Jul 30 08:24:45.616025 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.615965 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-online-tls\") pod \"7c24562d-b637-41de-bc75-77df719719f6\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " Jul 30 08:24:45.617866 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.617835 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-online-tls" (OuterVolumeSpecName: "online-tls") pod "7c24562d-b637-41de-bc75-77df719719f6" (UID: "7c24562d-b637-41de-bc75-77df719719f6"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:24:45.617968 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.617875 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c24562d-b637-41de-bc75-77df719719f6-kube-api-access-tt68k" (OuterVolumeSpecName: "kube-api-access-tt68k") pod "7c24562d-b637-41de-bc75-77df719719f6" (UID: "7c24562d-b637-41de-bc75-77df719719f6"). InnerVolumeSpecName "kube-api-access-tt68k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:24:45.617968 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.617910 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7c24562d-b637-41de-bc75-77df719719f6" (UID: "7c24562d-b637-41de-bc75-77df719719f6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:24:45.716917 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.716883 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7c24562d-b637-41de-bc75-77df719719f6-feast-data\") pod \"7c24562d-b637-41de-bc75-77df719719f6\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " Jul 30 08:24:45.717073 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.716971 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-offline-tls\") pod \"7c24562d-b637-41de-bc75-77df719719f6\" (UID: \"7c24562d-b637-41de-bc75-77df719719f6\") " Jul 30 08:24:45.717114 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.717088 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:24:45.717114 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.717103 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt68k\" (UniqueName: \"kubernetes.io/projected/7c24562d-b637-41de-bc75-77df719719f6-kube-api-access-tt68k\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:24:45.717114 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.717112 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:24:45.718920 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.718886 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7c24562d-b637-41de-bc75-77df719719f6" (UID: "7c24562d-b637-41de-bc75-77df719719f6"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:24:45.724869 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.724842 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c24562d-b637-41de-bc75-77df719719f6-feast-data" (OuterVolumeSpecName: "feast-data") pod "7c24562d-b637-41de-bc75-77df719719f6" (UID: "7c24562d-b637-41de-bc75-77df719719f6"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:24:45.817493 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.817430 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7c24562d-b637-41de-bc75-77df719719f6-offline-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:24:45.817493 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:45.817452 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7c24562d-b637-41de-bc75-77df719719f6-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:24:46.364452 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:46.364413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" event={"ID":"7c24562d-b637-41de-bc75-77df719719f6","Type":"ContainerDied","Data":"3aed4a1d0797be073fb90f0f84feb1496bf31fe761cc17b8ed67a41d7e7e9232"} Jul 30 08:24:46.364452 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:46.364443 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq" Jul 30 08:24:46.364950 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:46.364497 2576 scope.go:117] "RemoveContainer" containerID="a23a766f9fa9462111a9c9f2ca97b40ac327917844d0b0dce804dea076b86435" Jul 30 08:24:46.372503 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:46.372483 2576 scope.go:117] "RemoveContainer" containerID="1314be0b60020a686fa2361f3427069c25c29accd4728f05bc8e8942cc8e120b" Jul 30 08:24:46.379654 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:46.379634 2576 scope.go:117] "RemoveContainer" containerID="4d6541de1c45d23f47065672439d2fd03997d79ad39fd57a1991ef85619f3958" Jul 30 08:24:46.383563 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:46.383538 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq"] Jul 30 08:24:46.387164 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:46.387140 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-5969fbffbd-cz6rq"] Jul 30 08:24:46.387325 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:46.387298 2576 scope.go:117] "RemoveContainer" containerID="784bb3aae625f2f10e682315b20844bf51d87d13b6969520d123986f67d67d09" Jul 30 08:24:46.393794 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:46.393773 2576 scope.go:117] "RemoveContainer" containerID="a80af46e0b5206b305da7b2e98010d54e4f71744c2fe60db0773759fddf547d9" Jul 30 08:24:48.290558 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:24:48.290527 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c24562d-b637-41de-bc75-77df719719f6" path="/var/lib/kubelet/pods/7c24562d-b637-41de-bc75-77df719719f6/volumes" Jul 30 08:26:46.168144 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:46.168061 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:26:46.173164 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:46.173135 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:26:46.175041 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:46.175021 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:26:46.179817 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:46.179798 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:26:53.106927 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.106890 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-c5vv8"] Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107155 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="online" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107167 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="online" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107177 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="feast-apply" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107183 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="feast-apply" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107189 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73ce25c-f791-4966-b006-6a3cdc5302f1" containerName="postgres" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107195 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73ce25c-f791-4966-b006-6a3cdc5302f1" containerName="postgres" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107200 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="registry" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107205 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="registry" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107211 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="feast-init" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107216 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="feast-init" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107224 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="offline" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107230 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="offline" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107270 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="online" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107278 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73ce25c-f791-4966-b006-6a3cdc5302f1" containerName="postgres" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107284 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="registry" Jul 30 08:26:53.107289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.107290 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c24562d-b637-41de-bc75-77df719719f6" containerName="offline" Jul 30 08:26:53.110010 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.109993 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" Jul 30 08:26:53.112698 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.112670 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jul 30 08:26:53.112838 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.112717 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jul 30 08:26:53.113569 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.113551 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-m6z8g\"" Jul 30 08:26:53.127249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.127226 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-c5vv8"] Jul 30 08:26:53.232998 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.232962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jztsh\" (UniqueName: \"kubernetes.io/projected/36c01e33-447a-4a06-a058-5c647982ec9a-kube-api-access-jztsh\") pod \"redis-69bbd798b6-c5vv8\" (UID: \"36c01e33-447a-4a06-a058-5c647982ec9a\") " pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" Jul 30 08:26:53.334194 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.334155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jztsh\" (UniqueName: \"kubernetes.io/projected/36c01e33-447a-4a06-a058-5c647982ec9a-kube-api-access-jztsh\") pod \"redis-69bbd798b6-c5vv8\" (UID: \"36c01e33-447a-4a06-a058-5c647982ec9a\") " pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" Jul 30 08:26:53.342267 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.342246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jztsh\" (UniqueName: \"kubernetes.io/projected/36c01e33-447a-4a06-a058-5c647982ec9a-kube-api-access-jztsh\") pod \"redis-69bbd798b6-c5vv8\" (UID: \"36c01e33-447a-4a06-a058-5c647982ec9a\") " pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" Jul 30 08:26:53.436368 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.436278 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" Jul 30 08:26:53.560043 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.560013 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-c5vv8"] Jul 30 08:26:53.562532 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:26:53.562507 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36c01e33_447a_4a06_a058_5c647982ec9a.slice/crio-d06b3d3b11ba2d1ed04153e3358a9f4dae367f8b0d18c6995eed13aa7c28a65c WatchSource:0}: Error finding container d06b3d3b11ba2d1ed04153e3358a9f4dae367f8b0d18c6995eed13aa7c28a65c: Status 404 returned error can't find the container with id d06b3d3b11ba2d1ed04153e3358a9f4dae367f8b0d18c6995eed13aa7c28a65c Jul 30 08:26:53.826718 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:53.826686 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" event={"ID":"36c01e33-447a-4a06-a058-5c647982ec9a","Type":"ContainerStarted","Data":"d06b3d3b11ba2d1ed04153e3358a9f4dae367f8b0d18c6995eed13aa7c28a65c"} Jul 30 08:26:55.839639 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:55.839592 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" event={"ID":"36c01e33-447a-4a06-a058-5c647982ec9a","Type":"ContainerStarted","Data":"8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae"} Jul 30 08:26:55.855328 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:55.855271 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" podStartSLOduration=1.602440187 podStartE2EDuration="2.855255587s" totalImagesPullingTime="1.2528154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:26:55.854360591 +0000 UTC m=+910.114306926" watchObservedRunningTime="2026-07-30 08:26:55.855255587 +0000 UTC m=+910.115201921" Jul 30 08:26:58.530130 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.530100 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-9p8zh"] Jul 30 08:26:58.533225 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.533209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" Jul 30 08:26:58.536021 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.535999 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jul 30 08:26:58.541051 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.541026 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-9p8zh"] Jul 30 08:26:58.574098 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.574054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8n9\" (UniqueName: \"kubernetes.io/projected/45f60b91-0f69-41fc-a119-fbf8c4e451ef-kube-api-access-ch8n9\") pod \"postgres-864846d4f5-9p8zh\" (UID: \"45f60b91-0f69-41fc-a119-fbf8c4e451ef\") " pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" Jul 30 08:26:58.574263 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.574157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/45f60b91-0f69-41fc-a119-fbf8c4e451ef-postgresdata\") pod \"postgres-864846d4f5-9p8zh\" (UID: \"45f60b91-0f69-41fc-a119-fbf8c4e451ef\") " pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" Jul 30 08:26:58.675249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.675211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/45f60b91-0f69-41fc-a119-fbf8c4e451ef-postgresdata\") pod \"postgres-864846d4f5-9p8zh\" (UID: \"45f60b91-0f69-41fc-a119-fbf8c4e451ef\") " pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" Jul 30 08:26:58.675249 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.675256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8n9\" (UniqueName: \"kubernetes.io/projected/45f60b91-0f69-41fc-a119-fbf8c4e451ef-kube-api-access-ch8n9\") pod \"postgres-864846d4f5-9p8zh\" (UID: \"45f60b91-0f69-41fc-a119-fbf8c4e451ef\") " pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" Jul 30 08:26:58.675607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.675588 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/45f60b91-0f69-41fc-a119-fbf8c4e451ef-postgresdata\") pod \"postgres-864846d4f5-9p8zh\" (UID: \"45f60b91-0f69-41fc-a119-fbf8c4e451ef\") " pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" Jul 30 08:26:58.685336 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.685314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8n9\" (UniqueName: \"kubernetes.io/projected/45f60b91-0f69-41fc-a119-fbf8c4e451ef-kube-api-access-ch8n9\") pod \"postgres-864846d4f5-9p8zh\" (UID: \"45f60b91-0f69-41fc-a119-fbf8c4e451ef\") " pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" Jul 30 08:26:58.843206 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.843173 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" Jul 30 08:26:58.964054 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:58.964024 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-9p8zh"] Jul 30 08:26:58.966689 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:26:58.966663 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45f60b91_0f69_41fc_a119_fbf8c4e451ef.slice/crio-a3ed2835ddc5f3674c8d15d3a5a74a1a748c40bcbcb1810f10e2c14a53a2400a WatchSource:0}: Error finding container a3ed2835ddc5f3674c8d15d3a5a74a1a748c40bcbcb1810f10e2c14a53a2400a: Status 404 returned error can't find the container with id a3ed2835ddc5f3674c8d15d3a5a74a1a748c40bcbcb1810f10e2c14a53a2400a Jul 30 08:26:59.856429 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:59.856391 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" event={"ID":"45f60b91-0f69-41fc-a119-fbf8c4e451ef","Type":"ContainerStarted","Data":"c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c"} Jul 30 08:26:59.856788 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:59.856438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" event={"ID":"45f60b91-0f69-41fc-a119-fbf8c4e451ef","Type":"ContainerStarted","Data":"a3ed2835ddc5f3674c8d15d3a5a74a1a748c40bcbcb1810f10e2c14a53a2400a"} Jul 30 08:26:59.872899 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:26:59.872853 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" podStartSLOduration=1.872837729 podStartE2EDuration="1.872837729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:26:59.871679086 +0000 UTC m=+914.131625422" watchObservedRunningTime="2026-07-30 08:26:59.872837729 +0000 UTC m=+914.132784063" Jul 30 08:27:04.070362 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.070332 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v"] Jul 30 08:27:04.075950 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.075929 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.078477 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.078449 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jul 30 08:27:04.078626 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.078521 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jul 30 08:27:04.078626 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.078565 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jul 30 08:27:04.078762 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.078744 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-zqvph\"" Jul 30 08:27:04.086032 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.086009 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v"] Jul 30 08:27:04.221060 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.221024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-feast-data\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.221225 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.221133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-online-tls\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.221225 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.221172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k7t5\" (UniqueName: \"kubernetes.io/projected/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-kube-api-access-2k7t5\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.221225 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.221204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-registry-tls\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.322294 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.322218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-registry-tls\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.322294 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.322252 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-feast-data\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.322550 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.322309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-online-tls\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.322550 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.322324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k7t5\" (UniqueName: \"kubernetes.io/projected/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-kube-api-access-2k7t5\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.322550 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:27:04.322390 2576 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Jul 30 08:27:04.322550 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:27:04.322453 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-registry-tls podName:f8e27485-eec7-4d5a-8ec8-ff3768009ac7 nodeName:}" failed. No retries permitted until 2026-07-30 08:27:04.822435986 +0000 UTC m=+919.082382299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-registry-tls") pod "feast-credit-scoring-76bd7ccd4f-v8t4v" (UID: "f8e27485-eec7-4d5a-8ec8-ff3768009ac7") : secret "feast-credit-scoring-registry-tls" not found Jul 30 08:27:04.322782 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.322757 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-feast-data\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.324864 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.324844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-online-tls\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.330757 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.330736 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k7t5\" (UniqueName: \"kubernetes.io/projected/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-kube-api-access-2k7t5\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.827063 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.827029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-registry-tls\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.829297 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.829266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-registry-tls\") pod \"feast-credit-scoring-76bd7ccd4f-v8t4v\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:04.984702 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:04.984658 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:05.118839 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:05.118815 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v"] Jul 30 08:27:05.120982 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:27:05.120959 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e27485_eec7_4d5a_8ec8_ff3768009ac7.slice/crio-2b2c27476dd95e0b48b263c29746e3f2963b774a5e151d8c0dbf131b8dede6c0 WatchSource:0}: Error finding container 2b2c27476dd95e0b48b263c29746e3f2963b774a5e151d8c0dbf131b8dede6c0: Status 404 returned error can't find the container with id 2b2c27476dd95e0b48b263c29746e3f2963b774a5e151d8c0dbf131b8dede6c0 Jul 30 08:27:05.878977 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:05.878933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" event={"ID":"f8e27485-eec7-4d5a-8ec8-ff3768009ac7","Type":"ContainerStarted","Data":"df1eb0ff916dd9676de5094ea3a99c93138bc7b85dcd8a598f46c8cc0ff0b66b"} Jul 30 08:27:05.879184 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:05.879005 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" event={"ID":"f8e27485-eec7-4d5a-8ec8-ff3768009ac7","Type":"ContainerStarted","Data":"2b2c27476dd95e0b48b263c29746e3f2963b774a5e151d8c0dbf131b8dede6c0"} Jul 30 08:27:06.883964 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:06.883935 2576 generic.go:356] "Generic (PLEG): container finished" podID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerID="df1eb0ff916dd9676de5094ea3a99c93138bc7b85dcd8a598f46c8cc0ff0b66b" exitCode=0 Jul 30 08:27:06.883964 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:06.883966 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" event={"ID":"f8e27485-eec7-4d5a-8ec8-ff3768009ac7","Type":"ContainerDied","Data":"df1eb0ff916dd9676de5094ea3a99c93138bc7b85dcd8a598f46c8cc0ff0b66b"} Jul 30 08:27:07.889952 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:07.889914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" event={"ID":"f8e27485-eec7-4d5a-8ec8-ff3768009ac7","Type":"ContainerStarted","Data":"5ebaf208c08f28d8f4ff59fd07c7727a7b87727c899a2d3c0af763328c25f45a"} Jul 30 08:27:13.919706 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:13.919671 2576 generic.go:356] "Generic (PLEG): container finished" podID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerID="5ebaf208c08f28d8f4ff59fd07c7727a7b87727c899a2d3c0af763328c25f45a" exitCode=0 Jul 30 08:27:13.920172 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:13.919738 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" event={"ID":"f8e27485-eec7-4d5a-8ec8-ff3768009ac7","Type":"ContainerDied","Data":"5ebaf208c08f28d8f4ff59fd07c7727a7b87727c899a2d3c0af763328c25f45a"} Jul 30 08:27:14.926135 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:14.926104 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" event={"ID":"f8e27485-eec7-4d5a-8ec8-ff3768009ac7","Type":"ContainerStarted","Data":"18ee37b553f7aeda3f87f5b43e189e3933186833048715f7487dd49a00961e87"} Jul 30 08:27:14.926509 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:14.926161 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" event={"ID":"f8e27485-eec7-4d5a-8ec8-ff3768009ac7","Type":"ContainerStarted","Data":"6af539de0b70532a7b82e7728999a25a19e4ebb1be9c2ad1276d6f78383583e1"} Jul 30 08:27:14.960689 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:14.960636 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" podStartSLOduration=10.9606157 podStartE2EDuration="10.9606157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:27:14.958466416 +0000 UTC m=+929.218412779" watchObservedRunningTime="2026-07-30 08:27:14.9606157 +0000 UTC m=+929.220562037" Jul 30 08:27:14.985422 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:14.985395 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:14.985536 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:14.985451 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:16.985432 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:16.985388 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:16.985907 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:16.985453 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:16.986718 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:16.986687 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Jul 30 08:27:16.986831 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:16.986687 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 30 08:27:19.985540 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:19.985479 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Jul 30 08:27:19.991214 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:19.990723 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:19.994229 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:19.994206 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:22.985211 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:22.985168 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Jul 30 08:27:25.985624 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:25.985594 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:25.986406 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:25.986362 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:27:29.877072 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:29.877036 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf"] Jul 30 08:27:29.881512 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:29.881494 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:29.884103 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:29.884074 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jul 30 08:27:29.884243 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:29.884098 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jul 30 08:27:29.884243 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:29.884082 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-67h9w\"" Jul 30 08:27:29.884243 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:29.884132 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jul 30 08:27:29.891715 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:29.891695 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf"] Jul 30 08:27:30.012169 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.012149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a98aca69-0ccb-42b9-81f6-debe0b8112be-feast-data\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.012279 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.012197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-online-tls\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.012279 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.012218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgsl9\" (UniqueName: \"kubernetes.io/projected/a98aca69-0ccb-42b9-81f6-debe0b8112be-kube-api-access-fgsl9\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.012409 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.012305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-registry-tls\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.112695 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.112675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a98aca69-0ccb-42b9-81f6-debe0b8112be-feast-data\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.112772 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.112712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-online-tls\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.112772 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.112733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgsl9\" (UniqueName: \"kubernetes.io/projected/a98aca69-0ccb-42b9-81f6-debe0b8112be-kube-api-access-fgsl9\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.112772 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.112756 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-registry-tls\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.112913 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:27:30.112837 2576 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Jul 30 08:27:30.112913 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:27:30.112843 2576 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found Jul 30 08:27:30.112913 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:27:30.112910 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-registry-tls podName:a98aca69-0ccb-42b9-81f6-debe0b8112be nodeName:}" failed. No retries permitted until 2026-07-30 08:27:30.612893175 +0000 UTC m=+944.872839495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-registry-tls") pod "feast-driver-ranking-55d79f6b-jj7zf" (UID: "a98aca69-0ccb-42b9-81f6-debe0b8112be") : secret "feast-driver-ranking-registry-tls" not found Jul 30 08:27:30.113023 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:27:30.112929 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-online-tls podName:a98aca69-0ccb-42b9-81f6-debe0b8112be nodeName:}" failed. No retries permitted until 2026-07-30 08:27:30.612919716 +0000 UTC m=+944.872866032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-online-tls") pod "feast-driver-ranking-55d79f6b-jj7zf" (UID: "a98aca69-0ccb-42b9-81f6-debe0b8112be") : secret "feast-driver-ranking-online-tls" not found Jul 30 08:27:30.113062 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.113041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a98aca69-0ccb-42b9-81f6-debe0b8112be-feast-data\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.122049 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.122024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgsl9\" (UniqueName: \"kubernetes.io/projected/a98aca69-0ccb-42b9-81f6-debe0b8112be-kube-api-access-fgsl9\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.618205 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.618172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-online-tls\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.618364 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.618217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-registry-tls\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.620441 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.620419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-online-tls\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.620553 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.620484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-registry-tls\") pod \"feast-driver-ranking-55d79f6b-jj7zf\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.791224 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.791200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:30.921579 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.921547 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf"] Jul 30 08:27:30.925718 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:27:30.925694 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98aca69_0ccb_42b9_81f6_debe0b8112be.slice/crio-6fd17f73a1e1cddb7fe4ccf7752654bdb6784e5def21be9034fba3947876b1a4 WatchSource:0}: Error finding container 6fd17f73a1e1cddb7fe4ccf7752654bdb6784e5def21be9034fba3947876b1a4: Status 404 returned error can't find the container with id 6fd17f73a1e1cddb7fe4ccf7752654bdb6784e5def21be9034fba3947876b1a4 Jul 30 08:27:30.990851 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.990821 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" event={"ID":"a98aca69-0ccb-42b9-81f6-debe0b8112be","Type":"ContainerStarted","Data":"efd3778568dbc80ccbefa044b98aff6921c3b27a16854f0216786a77737ab576"} Jul 30 08:27:30.990957 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:30.990876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" event={"ID":"a98aca69-0ccb-42b9-81f6-debe0b8112be","Type":"ContainerStarted","Data":"6fd17f73a1e1cddb7fe4ccf7752654bdb6784e5def21be9034fba3947876b1a4"} Jul 30 08:27:36.012348 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:36.012315 2576 generic.go:356] "Generic (PLEG): container finished" podID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerID="efd3778568dbc80ccbefa044b98aff6921c3b27a16854f0216786a77737ab576" exitCode=0 Jul 30 08:27:36.012719 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:36.012403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" event={"ID":"a98aca69-0ccb-42b9-81f6-debe0b8112be","Type":"ContainerDied","Data":"efd3778568dbc80ccbefa044b98aff6921c3b27a16854f0216786a77737ab576"} Jul 30 08:27:37.018173 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:37.018139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" event={"ID":"a98aca69-0ccb-42b9-81f6-debe0b8112be","Type":"ContainerStarted","Data":"fcd4b678ceece7c07ea7c0ff53e76748f0504bf02f02b6812eb6cf654b79fcc0"} Jul 30 08:27:43.042174 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:43.042145 2576 generic.go:356] "Generic (PLEG): container finished" podID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerID="fcd4b678ceece7c07ea7c0ff53e76748f0504bf02f02b6812eb6cf654b79fcc0" exitCode=0 Jul 30 08:27:43.042528 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:43.042203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" event={"ID":"a98aca69-0ccb-42b9-81f6-debe0b8112be","Type":"ContainerDied","Data":"fcd4b678ceece7c07ea7c0ff53e76748f0504bf02f02b6812eb6cf654b79fcc0"} Jul 30 08:27:44.049164 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:44.049128 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" event={"ID":"a98aca69-0ccb-42b9-81f6-debe0b8112be","Type":"ContainerStarted","Data":"b0a1ccea710aa9330d43160739f91113a5f9423703b1c43c935665dbc9b7be7d"} Jul 30 08:27:44.049554 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:44.049187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" event={"ID":"a98aca69-0ccb-42b9-81f6-debe0b8112be","Type":"ContainerStarted","Data":"faf61fcc0c4cd180547afd653b570c3ae8f8c2be0ad2acadbab7af8455207ac2"} Jul 30 08:27:44.070529 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:44.070486 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" podStartSLOduration=15.070470057 podStartE2EDuration="15.070470057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:27:44.06828524 +0000 UTC m=+958.328231591" watchObservedRunningTime="2026-07-30 08:27:44.070470057 +0000 UTC m=+958.330416393" Jul 30 08:27:45.791637 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:45.791597 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:45.791996 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:45.791669 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:45.793318 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:45.793289 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Jul 30 08:27:45.793457 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:45.793288 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Jul 30 08:27:48.792138 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:48.792086 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Jul 30 08:27:49.104993 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:49.104902 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:49.105413 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:49.105391 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:49.108266 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:49.108247 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:50.792288 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:50.792251 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:51.792289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:51.792244 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Jul 30 08:27:54.792595 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:54.792561 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:27:54.793311 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:27:54.793294 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:29:07.132213 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.132182 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s"] Jul 30 08:29:07.132646 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.132408 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" podUID="88463ebb-d6e7-4acf-8bb8-51e81bd1ee66" containerName="manager" containerID="cri-o://f7810a97da7e5a4ad886fd1f7206ecf60e7bcf6e302efcc30b06b40e2982cc95" gracePeriod=10 Jul 30 08:29:07.377573 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.377542 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf"] Jul 30 08:29:07.377790 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.377769 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="registry" containerID="cri-o://faf61fcc0c4cd180547afd653b570c3ae8f8c2be0ad2acadbab7af8455207ac2" gracePeriod=30 Jul 30 08:29:07.378031 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.377911 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="online" containerID="cri-o://b0a1ccea710aa9330d43160739f91113a5f9423703b1c43c935665dbc9b7be7d" gracePeriod=30 Jul 30 08:29:07.381885 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.381338 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v"] Jul 30 08:29:07.381885 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.381579 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="registry" containerID="cri-o://6af539de0b70532a7b82e7728999a25a19e4ebb1be9c2ad1276d6f78383583e1" gracePeriod=30 Jul 30 08:29:07.381885 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.381788 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="online" containerID="cri-o://18ee37b553f7aeda3f87f5b43e189e3933186833048715f7487dd49a00961e87" gracePeriod=30 Jul 30 08:29:07.399408 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.399243 2576 generic.go:356] "Generic (PLEG): container finished" podID="88463ebb-d6e7-4acf-8bb8-51e81bd1ee66" containerID="f7810a97da7e5a4ad886fd1f7206ecf60e7bcf6e302efcc30b06b40e2982cc95" exitCode=0 Jul 30 08:29:07.399408 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.399300 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" event={"ID":"88463ebb-d6e7-4acf-8bb8-51e81bd1ee66","Type":"ContainerDied","Data":"f7810a97da7e5a4ad886fd1f7206ecf60e7bcf6e302efcc30b06b40e2982cc95"} Jul 30 08:29:07.408330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.408310 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" Jul 30 08:29:07.543240 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.543213 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7vp9\" (UniqueName: \"kubernetes.io/projected/88463ebb-d6e7-4acf-8bb8-51e81bd1ee66-kube-api-access-r7vp9\") pod \"88463ebb-d6e7-4acf-8bb8-51e81bd1ee66\" (UID: \"88463ebb-d6e7-4acf-8bb8-51e81bd1ee66\") " Jul 30 08:29:07.545190 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.545162 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88463ebb-d6e7-4acf-8bb8-51e81bd1ee66-kube-api-access-r7vp9" (OuterVolumeSpecName: "kube-api-access-r7vp9") pod "88463ebb-d6e7-4acf-8bb8-51e81bd1ee66" (UID: "88463ebb-d6e7-4acf-8bb8-51e81bd1ee66"). InnerVolumeSpecName "kube-api-access-r7vp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:29:07.644505 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:07.644442 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7vp9\" (UniqueName: \"kubernetes.io/projected/88463ebb-d6e7-4acf-8bb8-51e81bd1ee66-kube-api-access-r7vp9\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:08.405130 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:08.405073 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" Jul 30 08:29:08.405130 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:08.405082 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s" event={"ID":"88463ebb-d6e7-4acf-8bb8-51e81bd1ee66","Type":"ContainerDied","Data":"950a95cb0f022bce3163180c936d0a1213acb71d4bcfa73c60f3154435cb809f"} Jul 30 08:29:08.405597 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:08.405148 2576 scope.go:117] "RemoveContainer" containerID="f7810a97da7e5a4ad886fd1f7206ecf60e7bcf6e302efcc30b06b40e2982cc95" Jul 30 08:29:08.408037 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:08.408015 2576 generic.go:356] "Generic (PLEG): container finished" podID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerID="b0a1ccea710aa9330d43160739f91113a5f9423703b1c43c935665dbc9b7be7d" exitCode=0 Jul 30 08:29:08.408143 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:08.408094 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" event={"ID":"a98aca69-0ccb-42b9-81f6-debe0b8112be","Type":"ContainerDied","Data":"b0a1ccea710aa9330d43160739f91113a5f9423703b1c43c935665dbc9b7be7d"} Jul 30 08:29:08.410695 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:08.410675 2576 generic.go:356] "Generic (PLEG): container finished" podID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerID="18ee37b553f7aeda3f87f5b43e189e3933186833048715f7487dd49a00961e87" exitCode=0 Jul 30 08:29:08.410796 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:08.410745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" event={"ID":"f8e27485-eec7-4d5a-8ec8-ff3768009ac7","Type":"ContainerDied","Data":"18ee37b553f7aeda3f87f5b43e189e3933186833048715f7487dd49a00961e87"} Jul 30 08:29:08.422536 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:08.422514 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s"] Jul 30 08:29:08.426357 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:08.426337 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-67978c95f8-5sp7s"] Jul 30 08:29:09.105990 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:09.105955 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Jul 30 08:29:09.670936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:09.670898 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-9p8zh"] Jul 30 08:29:09.671333 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:09.671062 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" podUID="45f60b91-0f69-41fc-a119-fbf8c4e451ef" containerName="postgres" containerID="cri-o://c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c" gracePeriod=30 Jul 30 08:29:09.676295 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:09.676264 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-c5vv8"] Jul 30 08:29:09.676514 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:09.676473 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" podUID="36c01e33-447a-4a06-a058-5c647982ec9a" containerName="redis" containerID="cri-o://8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae" gracePeriod=30 Jul 30 08:29:09.935289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:09.935268 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" Jul 30 08:29:09.937814 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:09.937798 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" Jul 30 08:29:09.995396 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:09.995343 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 30 08:29:10.064881 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.064851 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch8n9\" (UniqueName: \"kubernetes.io/projected/45f60b91-0f69-41fc-a119-fbf8c4e451ef-kube-api-access-ch8n9\") pod \"45f60b91-0f69-41fc-a119-fbf8c4e451ef\" (UID: \"45f60b91-0f69-41fc-a119-fbf8c4e451ef\") " Jul 30 08:29:10.065024 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.064926 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/45f60b91-0f69-41fc-a119-fbf8c4e451ef-postgresdata\") pod \"45f60b91-0f69-41fc-a119-fbf8c4e451ef\" (UID: \"45f60b91-0f69-41fc-a119-fbf8c4e451ef\") " Jul 30 08:29:10.065024 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.064980 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jztsh\" (UniqueName: \"kubernetes.io/projected/36c01e33-447a-4a06-a058-5c647982ec9a-kube-api-access-jztsh\") pod \"36c01e33-447a-4a06-a058-5c647982ec9a\" (UID: \"36c01e33-447a-4a06-a058-5c647982ec9a\") " Jul 30 08:29:10.066969 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.066940 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f60b91-0f69-41fc-a119-fbf8c4e451ef-kube-api-access-ch8n9" (OuterVolumeSpecName: "kube-api-access-ch8n9") pod "45f60b91-0f69-41fc-a119-fbf8c4e451ef" (UID: "45f60b91-0f69-41fc-a119-fbf8c4e451ef"). InnerVolumeSpecName "kube-api-access-ch8n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:29:10.066969 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.066959 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c01e33-447a-4a06-a058-5c647982ec9a-kube-api-access-jztsh" (OuterVolumeSpecName: "kube-api-access-jztsh") pod "36c01e33-447a-4a06-a058-5c647982ec9a" (UID: "36c01e33-447a-4a06-a058-5c647982ec9a"). InnerVolumeSpecName "kube-api-access-jztsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:29:10.092844 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.092819 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f60b91-0f69-41fc-a119-fbf8c4e451ef-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "45f60b91-0f69-41fc-a119-fbf8c4e451ef" (UID: "45f60b91-0f69-41fc-a119-fbf8c4e451ef"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:29:10.166076 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.166056 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jztsh\" (UniqueName: \"kubernetes.io/projected/36c01e33-447a-4a06-a058-5c647982ec9a-kube-api-access-jztsh\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:10.166076 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.166076 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch8n9\" (UniqueName: \"kubernetes.io/projected/45f60b91-0f69-41fc-a119-fbf8c4e451ef-kube-api-access-ch8n9\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:10.166178 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.166106 2576 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/45f60b91-0f69-41fc-a119-fbf8c4e451ef-postgresdata\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:10.291292 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.291265 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88463ebb-d6e7-4acf-8bb8-51e81bd1ee66" path="/var/lib/kubelet/pods/88463ebb-d6e7-4acf-8bb8-51e81bd1ee66/volumes" Jul 30 08:29:10.421070 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.421051 2576 generic.go:356] "Generic (PLEG): container finished" podID="36c01e33-447a-4a06-a058-5c647982ec9a" containerID="8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae" exitCode=0 Jul 30 08:29:10.421161 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.421101 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" Jul 30 08:29:10.421161 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.421114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" event={"ID":"36c01e33-447a-4a06-a058-5c647982ec9a","Type":"ContainerDied","Data":"8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae"} Jul 30 08:29:10.421161 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.421139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-c5vv8" event={"ID":"36c01e33-447a-4a06-a058-5c647982ec9a","Type":"ContainerDied","Data":"d06b3d3b11ba2d1ed04153e3358a9f4dae367f8b0d18c6995eed13aa7c28a65c"} Jul 30 08:29:10.421161 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.421161 2576 scope.go:117] "RemoveContainer" containerID="8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae" Jul 30 08:29:10.422351 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.422328 2576 generic.go:356] "Generic (PLEG): container finished" podID="45f60b91-0f69-41fc-a119-fbf8c4e451ef" containerID="c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c" exitCode=0 Jul 30 08:29:10.422438 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.422368 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" Jul 30 08:29:10.422438 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.422417 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" event={"ID":"45f60b91-0f69-41fc-a119-fbf8c4e451ef","Type":"ContainerDied","Data":"c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c"} Jul 30 08:29:10.422545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.422464 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-9p8zh" event={"ID":"45f60b91-0f69-41fc-a119-fbf8c4e451ef","Type":"ContainerDied","Data":"a3ed2835ddc5f3674c8d15d3a5a74a1a748c40bcbcb1810f10e2c14a53a2400a"} Jul 30 08:29:10.428974 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.428959 2576 scope.go:117] "RemoveContainer" containerID="8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae" Jul 30 08:29:10.429210 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:29:10.429192 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae\": container with ID starting with 8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae not found: ID does not exist" containerID="8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae" Jul 30 08:29:10.429258 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.429217 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae"} err="failed to get container status \"8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae\": rpc error: code = NotFound desc = could not find container \"8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae\": container with ID starting with 8ca54239c07c96374667ffb1e9d4bf82918cccb2511813be8482a9511338b6ae not found: ID does not exist" Jul 30 08:29:10.429258 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.429232 2576 scope.go:117] "RemoveContainer" containerID="c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c" Jul 30 08:29:10.435999 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.435976 2576 scope.go:117] "RemoveContainer" containerID="c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c" Jul 30 08:29:10.436259 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:29:10.436241 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c\": container with ID starting with c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c not found: ID does not exist" containerID="c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c" Jul 30 08:29:10.436313 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.436266 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c"} err="failed to get container status \"c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c\": rpc error: code = NotFound desc = could not find container \"c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c\": container with ID starting with c14ea34974c5bee16de7b8a8598701209078904a96bcbf8401ec0b490ed7f99c not found: ID does not exist" Jul 30 08:29:10.436616 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.436600 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-c5vv8"] Jul 30 08:29:10.441527 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.441507 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-c5vv8"] Jul 30 08:29:10.452748 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.452722 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-9p8zh"] Jul 30 08:29:10.456281 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:10.456262 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-9p8zh"] Jul 30 08:29:12.290617 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:12.290590 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36c01e33-447a-4a06-a058-5c647982ec9a" path="/var/lib/kubelet/pods/36c01e33-447a-4a06-a058-5c647982ec9a/volumes" Jul 30 08:29:12.290938 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:12.290911 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45f60b91-0f69-41fc-a119-fbf8c4e451ef" path="/var/lib/kubelet/pods/45f60b91-0f69-41fc-a119-fbf8c4e451ef/volumes" Jul 30 08:29:19.105576 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:19.105532 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Jul 30 08:29:19.991723 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:19.991685 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 30 08:29:29.106490 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:29.106447 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Jul 30 08:29:29.106879 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:29.106574 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:29:29.991530 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:29.991491 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 30 08:29:29.991749 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:29.991582 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:29:37.528745 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:37.528718 2576 generic.go:356] "Generic (PLEG): container finished" podID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerID="6af539de0b70532a7b82e7728999a25a19e4ebb1be9c2ad1276d6f78383583e1" exitCode=137 Jul 30 08:29:37.529091 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:37.528791 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" event={"ID":"f8e27485-eec7-4d5a-8ec8-ff3768009ac7","Type":"ContainerDied","Data":"6af539de0b70532a7b82e7728999a25a19e4ebb1be9c2ad1276d6f78383583e1"} Jul 30 08:29:37.531112 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:37.531068 2576 generic.go:356] "Generic (PLEG): container finished" podID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerID="faf61fcc0c4cd180547afd653b570c3ae8f8c2be0ad2acadbab7af8455207ac2" exitCode=137 Jul 30 08:29:37.531112 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:37.531102 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" event={"ID":"a98aca69-0ccb-42b9-81f6-debe0b8112be","Type":"ContainerDied","Data":"faf61fcc0c4cd180547afd653b570c3ae8f8c2be0ad2acadbab7af8455207ac2"} Jul 30 08:29:38.051986 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.051961 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:29:38.054256 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.054237 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:29:38.155051 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.154991 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgsl9\" (UniqueName: \"kubernetes.io/projected/a98aca69-0ccb-42b9-81f6-debe0b8112be-kube-api-access-fgsl9\") pod \"a98aca69-0ccb-42b9-81f6-debe0b8112be\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " Jul 30 08:29:38.155051 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.155025 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-registry-tls\") pod \"a98aca69-0ccb-42b9-81f6-debe0b8112be\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " Jul 30 08:29:38.155051 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.155045 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a98aca69-0ccb-42b9-81f6-debe0b8112be-feast-data\") pod \"a98aca69-0ccb-42b9-81f6-debe0b8112be\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " Jul 30 08:29:38.155266 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.155065 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-online-tls\") pod \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " Jul 30 08:29:38.155266 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.155092 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-registry-tls\") pod \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " Jul 30 08:29:38.155266 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.155132 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k7t5\" (UniqueName: \"kubernetes.io/projected/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-kube-api-access-2k7t5\") pod \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " Jul 30 08:29:38.155266 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.155147 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-online-tls\") pod \"a98aca69-0ccb-42b9-81f6-debe0b8112be\" (UID: \"a98aca69-0ccb-42b9-81f6-debe0b8112be\") " Jul 30 08:29:38.155266 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.155177 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-feast-data\") pod \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\" (UID: \"f8e27485-eec7-4d5a-8ec8-ff3768009ac7\") " Jul 30 08:29:38.155772 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.155738 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a98aca69-0ccb-42b9-81f6-debe0b8112be-feast-data" (OuterVolumeSpecName: "feast-data") pod "a98aca69-0ccb-42b9-81f6-debe0b8112be" (UID: "a98aca69-0ccb-42b9-81f6-debe0b8112be"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:29:38.157415 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.157358 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-online-tls" (OuterVolumeSpecName: "online-tls") pod "a98aca69-0ccb-42b9-81f6-debe0b8112be" (UID: "a98aca69-0ccb-42b9-81f6-debe0b8112be"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:29:38.157555 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.157524 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f8e27485-eec7-4d5a-8ec8-ff3768009ac7" (UID: "f8e27485-eec7-4d5a-8ec8-ff3768009ac7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:29:38.157654 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.157593 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-online-tls" (OuterVolumeSpecName: "online-tls") pod "f8e27485-eec7-4d5a-8ec8-ff3768009ac7" (UID: "f8e27485-eec7-4d5a-8ec8-ff3768009ac7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:29:38.157654 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.157601 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a98aca69-0ccb-42b9-81f6-debe0b8112be" (UID: "a98aca69-0ccb-42b9-81f6-debe0b8112be"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:29:38.157654 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.157622 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98aca69-0ccb-42b9-81f6-debe0b8112be-kube-api-access-fgsl9" (OuterVolumeSpecName: "kube-api-access-fgsl9") pod "a98aca69-0ccb-42b9-81f6-debe0b8112be" (UID: "a98aca69-0ccb-42b9-81f6-debe0b8112be"). InnerVolumeSpecName "kube-api-access-fgsl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:29:38.157867 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.157850 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-kube-api-access-2k7t5" (OuterVolumeSpecName: "kube-api-access-2k7t5") pod "f8e27485-eec7-4d5a-8ec8-ff3768009ac7" (UID: "f8e27485-eec7-4d5a-8ec8-ff3768009ac7"). InnerVolumeSpecName "kube-api-access-2k7t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:29:38.164038 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.164017 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-feast-data" (OuterVolumeSpecName: "feast-data") pod "f8e27485-eec7-4d5a-8ec8-ff3768009ac7" (UID: "f8e27485-eec7-4d5a-8ec8-ff3768009ac7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:29:38.256454 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.256436 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:38.256454 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.256452 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a98aca69-0ccb-42b9-81f6-debe0b8112be-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:38.256563 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.256460 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:38.256563 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.256468 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:38.256563 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.256476 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k7t5\" (UniqueName: \"kubernetes.io/projected/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-kube-api-access-2k7t5\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:38.256563 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.256484 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a98aca69-0ccb-42b9-81f6-debe0b8112be-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:38.256563 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.256492 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e27485-eec7-4d5a-8ec8-ff3768009ac7-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:38.256563 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.256500 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgsl9\" (UniqueName: \"kubernetes.io/projected/a98aca69-0ccb-42b9-81f6-debe0b8112be-kube-api-access-fgsl9\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:29:38.537786 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.537732 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" Jul 30 08:29:38.537786 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.537730 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v" event={"ID":"f8e27485-eec7-4d5a-8ec8-ff3768009ac7","Type":"ContainerDied","Data":"2b2c27476dd95e0b48b263c29746e3f2963b774a5e151d8c0dbf131b8dede6c0"} Jul 30 08:29:38.538194 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.537820 2576 scope.go:117] "RemoveContainer" containerID="18ee37b553f7aeda3f87f5b43e189e3933186833048715f7487dd49a00961e87" Jul 30 08:29:38.540642 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.540617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" event={"ID":"a98aca69-0ccb-42b9-81f6-debe0b8112be","Type":"ContainerDied","Data":"6fd17f73a1e1cddb7fe4ccf7752654bdb6784e5def21be9034fba3947876b1a4"} Jul 30 08:29:38.540745 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.540675 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf" Jul 30 08:29:38.547166 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.547145 2576 scope.go:117] "RemoveContainer" containerID="6af539de0b70532a7b82e7728999a25a19e4ebb1be9c2ad1276d6f78383583e1" Jul 30 08:29:38.555052 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.555000 2576 scope.go:117] "RemoveContainer" containerID="5ebaf208c08f28d8f4ff59fd07c7727a7b87727c899a2d3c0af763328c25f45a" Jul 30 08:29:38.555515 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.555490 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v"] Jul 30 08:29:38.559510 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.559486 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-76bd7ccd4f-v8t4v"] Jul 30 08:29:38.562900 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.562882 2576 scope.go:117] "RemoveContainer" containerID="df1eb0ff916dd9676de5094ea3a99c93138bc7b85dcd8a598f46c8cc0ff0b66b" Jul 30 08:29:38.569753 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.569731 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf"] Jul 30 08:29:38.569889 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.569878 2576 scope.go:117] "RemoveContainer" containerID="b0a1ccea710aa9330d43160739f91113a5f9423703b1c43c935665dbc9b7be7d" Jul 30 08:29:38.575401 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.575364 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-55d79f6b-jj7zf"] Jul 30 08:29:38.576781 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.576768 2576 scope.go:117] "RemoveContainer" containerID="faf61fcc0c4cd180547afd653b570c3ae8f8c2be0ad2acadbab7af8455207ac2" Jul 30 08:29:38.583642 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.583624 2576 scope.go:117] "RemoveContainer" containerID="fcd4b678ceece7c07ea7c0ff53e76748f0504bf02f02b6812eb6cf654b79fcc0" Jul 30 08:29:38.590116 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:38.590099 2576 scope.go:117] "RemoveContainer" containerID="efd3778568dbc80ccbefa044b98aff6921c3b27a16854f0216786a77737ab576" Jul 30 08:29:40.291345 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:40.291308 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" path="/var/lib/kubelet/pods/a98aca69-0ccb-42b9-81f6-debe0b8112be/volumes" Jul 30 08:29:40.291826 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:40.291809 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" path="/var/lib/kubelet/pods/f8e27485-eec7-4d5a-8ec8-ff3768009ac7/volumes" Jul 30 08:29:45.221274 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221243 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz"] Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221489 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="feast-init" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221499 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="feast-init" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221509 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c01e33-447a-4a06-a058-5c647982ec9a" containerName="redis" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221516 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c01e33-447a-4a06-a058-5c647982ec9a" containerName="redis" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221522 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="online" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221527 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="online" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221534 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88463ebb-d6e7-4acf-8bb8-51e81bd1ee66" containerName="manager" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221539 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88463ebb-d6e7-4acf-8bb8-51e81bd1ee66" containerName="manager" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221545 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="registry" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221550 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="registry" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221555 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="online" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221560 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="online" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221565 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f60b91-0f69-41fc-a119-fbf8c4e451ef" containerName="postgres" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221570 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f60b91-0f69-41fc-a119-fbf8c4e451ef" containerName="postgres" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221576 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="feast-apply" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221581 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="feast-apply" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221586 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="registry" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221591 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="registry" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221602 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="feast-init" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221606 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="feast-init" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221612 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="feast-apply" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221616 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="feast-apply" Jul 30 08:29:45.221661 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221670 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="45f60b91-0f69-41fc-a119-fbf8c4e451ef" containerName="postgres" Jul 30 08:29:45.222312 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221678 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="online" Jul 30 08:29:45.222312 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221685 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="registry" Jul 30 08:29:45.222312 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221690 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="36c01e33-447a-4a06-a058-5c647982ec9a" containerName="redis" Jul 30 08:29:45.222312 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221696 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88463ebb-d6e7-4acf-8bb8-51e81bd1ee66" containerName="manager" Jul 30 08:29:45.222312 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221700 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98aca69-0ccb-42b9-81f6-debe0b8112be" containerName="registry" Jul 30 08:29:45.222312 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.221705 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e27485-eec7-4d5a-8ec8-ff3768009ac7" containerName="online" Jul 30 08:29:45.224675 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.224658 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.227059 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.227039 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 30 08:29:45.227189 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.227146 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 30 08:29:45.227279 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.227261 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 30 08:29:45.227413 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.227398 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 30 08:29:45.227938 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.227695 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 30 08:29:45.227938 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.227515 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 30 08:29:45.228085 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.227964 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-rkbth\"" Jul 30 08:29:45.236015 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.235981 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz"] Jul 30 08:29:45.300220 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.300198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.300308 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.300229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.300308 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.300254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbxl\" (UniqueName: \"kubernetes.io/projected/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-kube-api-access-ztbxl\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.300308 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.300291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.300435 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.300318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.300435 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.300361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.401519 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.401497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.401607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.401532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.401607 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.401554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.401676 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:29:45.401623 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 30 08:29:45.401676 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:29:45.401676 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-registry-tls podName:30cdde88-0b6b-4f7e-bdaa-3b547c7352ba nodeName:}" failed. No retries permitted until 2026-07-30 08:29:45.901661021 +0000 UTC m=+1080.161607334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-registry-tls") pod "feast-simple-feast-setup-777d6cc998-pbpdz" (UID: "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba") : secret "feast-simple-feast-setup-registry-tls" not found Jul 30 08:29:45.401742 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.401687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbxl\" (UniqueName: \"kubernetes.io/projected/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-kube-api-access-ztbxl\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.401742 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.401705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.401742 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.401718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.401844 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:29:45.401824 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:29:45.401915 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:29:45.401902 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-offline-tls podName:30cdde88-0b6b-4f7e-bdaa-3b547c7352ba nodeName:}" failed. No retries permitted until 2026-07-30 08:29:45.901882129 +0000 UTC m=+1080.161828456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-offline-tls") pod "feast-simple-feast-setup-777d6cc998-pbpdz" (UID: "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba") : secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:29:45.402040 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.402024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.403912 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.403883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.403996 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.403966 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.410565 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.410543 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbxl\" (UniqueName: \"kubernetes.io/projected/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-kube-api-access-ztbxl\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.905248 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.905220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.905362 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.905255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.907504 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.907473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:45.907613 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:45.907593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-pbpdz\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:46.151139 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:46.151119 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-rkbth\"" Jul 30 08:29:46.159399 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:46.159346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:29:46.291052 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:46.291028 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz"] Jul 30 08:29:46.291549 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:29:46.291528 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30cdde88_0b6b_4f7e_bdaa_3b547c7352ba.slice/crio-3be8946501e5f509591b85ae5a89a8d26b721684d4b58e59ea084919c44e76bb WatchSource:0}: Error finding container 3be8946501e5f509591b85ae5a89a8d26b721684d4b58e59ea084919c44e76bb: Status 404 returned error can't find the container with id 3be8946501e5f509591b85ae5a89a8d26b721684d4b58e59ea084919c44e76bb Jul 30 08:29:46.292981 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:46.292966 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 08:29:46.571653 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:29:46.571619 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerStarted","Data":"3be8946501e5f509591b85ae5a89a8d26b721684d4b58e59ea084919c44e76bb"} Jul 30 08:30:00.630687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:00.630644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerStarted","Data":"9b43a541ac9404e1ac4f686431a57188d437bafa2daff3429e26c9f2b6aa3f94"} Jul 30 08:30:04.646220 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:04.646137 2576 generic.go:356] "Generic (PLEG): container finished" podID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerID="9b43a541ac9404e1ac4f686431a57188d437bafa2daff3429e26c9f2b6aa3f94" exitCode=0 Jul 30 08:30:04.646220 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:04.646210 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerDied","Data":"9b43a541ac9404e1ac4f686431a57188d437bafa2daff3429e26c9f2b6aa3f94"} Jul 30 08:30:05.651331 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:05.651300 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerStarted","Data":"324d3e3afd94321a0538ab68a2e5d405784fbc0c11e61f93f627b562473fecac"} Jul 30 08:30:10.669874 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:10.669839 2576 generic.go:356] "Generic (PLEG): container finished" podID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerID="324d3e3afd94321a0538ab68a2e5d405784fbc0c11e61f93f627b562473fecac" exitCode=0 Jul 30 08:30:10.670265 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:10.669888 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerDied","Data":"324d3e3afd94321a0538ab68a2e5d405784fbc0c11e61f93f627b562473fecac"} Jul 30 08:30:11.679197 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:11.679150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerStarted","Data":"64d7dc83209f39324344ad500189efaa690e3d7466f5a601aabd504f691cc7b3"} Jul 30 08:30:11.679681 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:11.679210 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerStarted","Data":"6d1735d4841103cb609eff748ea133b74f3e7bacf12d73bc6c153ee8b02b0829"} Jul 30 08:30:11.679681 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:11.679232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerStarted","Data":"714ff9961d79ee249be0aeb14214e708dfc0d327748a3ee0095070015be11e29"} Jul 30 08:30:11.679681 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:11.679250 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerStarted","Data":"8d1e379c9bdbdf0744104fc4387f6d4f977d28c75232e524369231352fae41f2"} Jul 30 08:30:11.704451 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:11.704395 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podStartSLOduration=12.985676321 podStartE2EDuration="26.704357864s" totalImagesPullingTime="13.718681543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:30:11.702441926 +0000 UTC m=+1105.962388286" watchObservedRunningTime="2026-07-30 08:30:11.704357864 +0000 UTC m=+1105.964304197" Jul 30 08:30:13.160561 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:13.159983 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:13.160995 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:13.160602 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:13.160995 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:13.160628 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:13.160995 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:13.160647 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:13.162086 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:13.162055 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Jul 30 08:30:13.162313 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:13.162077 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Jul 30 08:30:13.162421 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:13.162103 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 30 08:30:13.162421 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:13.162138 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 30 08:30:16.160078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:16.159721 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:16.160078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:16.159828 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:16.160078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:16.159849 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:16.160078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:16.159933 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:16.160078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:16.159964 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 30 08:30:16.160078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:16.159995 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Jul 30 08:30:16.160078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:16.159987 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 30 08:30:16.161065 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:16.161047 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:16.707989 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:16.707959 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:19.160615 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:19.160587 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:19.161021 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:19.160777 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:19.165401 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:19.165359 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:19.718846 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:19.718824 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:19.718993 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:19.718887 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:30:19.721245 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:30:19.721230 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:31:21.899614 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:21.899517 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz"] Jul 30 08:31:21.900209 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:21.899804 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="registry" containerID="cri-o://8d1e379c9bdbdf0744104fc4387f6d4f977d28c75232e524369231352fae41f2" gracePeriod=30 Jul 30 08:31:21.900209 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:21.899845 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="offline" containerID="cri-o://6d1735d4841103cb609eff748ea133b74f3e7bacf12d73bc6c153ee8b02b0829" gracePeriod=30 Jul 30 08:31:21.900209 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:21.899862 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="online" containerID="cri-o://714ff9961d79ee249be0aeb14214e708dfc0d327748a3ee0095070015be11e29" gracePeriod=30 Jul 30 08:31:21.900209 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:21.899973 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="ui" containerID="cri-o://64d7dc83209f39324344ad500189efaa690e3d7466f5a601aabd504f691cc7b3" gracePeriod=30 Jul 30 08:31:22.159423 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.159327 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp"] Jul 30 08:31:22.162337 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.162318 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.164673 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.164656 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-n2fzt\"" Jul 30 08:31:22.173970 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.173944 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp"] Jul 30 08:31:22.179161 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.179134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.179293 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.179177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9f60a33c-ff9d-400a-a75c-88ddde4fb480-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.179293 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.179207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.179416 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.179334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.179598 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.179575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.179656 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.179620 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjrj\" (UniqueName: \"kubernetes.io/projected/9f60a33c-ff9d-400a-a75c-88ddde4fb480-kube-api-access-9qjrj\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.280628 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.280594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9f60a33c-ff9d-400a-a75c-88ddde4fb480-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.280628 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.280628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.280885 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.280679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.280885 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.280694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.280885 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.280716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjrj\" (UniqueName: \"kubernetes.io/projected/9f60a33c-ff9d-400a-a75c-88ddde4fb480-kube-api-access-9qjrj\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.280885 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.280751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.280885 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:31:22.280829 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:31:22.281116 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:31:22.280902 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-offline-tls podName:9f60a33c-ff9d-400a-a75c-88ddde4fb480 nodeName:}" failed. No retries permitted until 2026-07-30 08:31:22.780880369 +0000 UTC m=+1177.040826682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-offline-tls") pod "feast-simple-feast-setup-777d6cc998-wpgrp" (UID: "9f60a33c-ff9d-400a-a75c-88ddde4fb480") : secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:31:22.281116 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:31:22.281053 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 30 08:31:22.281116 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.281067 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9f60a33c-ff9d-400a-a75c-88ddde4fb480-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.281116 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:31:22.281094 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-online-tls podName:9f60a33c-ff9d-400a-a75c-88ddde4fb480 nodeName:}" failed. No retries permitted until 2026-07-30 08:31:22.781080874 +0000 UTC m=+1177.041027193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-online-tls") pod "feast-simple-feast-setup-777d6cc998-wpgrp" (UID: "9f60a33c-ff9d-400a-a75c-88ddde4fb480") : secret "feast-simple-feast-setup-online-tls" not found Jul 30 08:31:22.283397 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.283344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.283527 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.283456 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.288351 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.288324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjrj\" (UniqueName: \"kubernetes.io/projected/9f60a33c-ff9d-400a-a75c-88ddde4fb480-kube-api-access-9qjrj\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.782890 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.782817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.782890 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.782857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.784994 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.784972 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.785091 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.785074 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-wpgrp\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:22.939464 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.939432 2576 generic.go:356] "Generic (PLEG): container finished" podID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerID="64d7dc83209f39324344ad500189efaa690e3d7466f5a601aabd504f691cc7b3" exitCode=0 Jul 30 08:31:22.939464 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.939455 2576 generic.go:356] "Generic (PLEG): container finished" podID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerID="714ff9961d79ee249be0aeb14214e708dfc0d327748a3ee0095070015be11e29" exitCode=0 Jul 30 08:31:22.939800 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.939506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerDied","Data":"64d7dc83209f39324344ad500189efaa690e3d7466f5a601aabd504f691cc7b3"} Jul 30 08:31:22.939800 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:22.939545 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerDied","Data":"714ff9961d79ee249be0aeb14214e708dfc0d327748a3ee0095070015be11e29"} Jul 30 08:31:23.071501 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:23.071476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:23.197440 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:23.197418 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp"] Jul 30 08:31:23.198642 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:31:23.198616 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f60a33c_ff9d_400a_a75c_88ddde4fb480.slice/crio-627e1fcd7f2e25e0a79f160963d9698c6adcc5435d27398530f72d3b5c6609b7 WatchSource:0}: Error finding container 627e1fcd7f2e25e0a79f160963d9698c6adcc5435d27398530f72d3b5c6609b7: Status 404 returned error can't find the container with id 627e1fcd7f2e25e0a79f160963d9698c6adcc5435d27398530f72d3b5c6609b7 Jul 30 08:31:23.945267 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:23.945234 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerStarted","Data":"72b25d770ba303516e7cafcff2460ae48a3eb45825d785be4533693a0d12342d"} Jul 30 08:31:23.945703 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:23.945279 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerStarted","Data":"627e1fcd7f2e25e0a79f160963d9698c6adcc5435d27398530f72d3b5c6609b7"} Jul 30 08:31:27.960225 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:27.960195 2576 generic.go:356] "Generic (PLEG): container finished" podID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerID="72b25d770ba303516e7cafcff2460ae48a3eb45825d785be4533693a0d12342d" exitCode=0 Jul 30 08:31:27.960576 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:27.960253 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerDied","Data":"72b25d770ba303516e7cafcff2460ae48a3eb45825d785be4533693a0d12342d"} Jul 30 08:31:28.965430 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:28.965398 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerStarted","Data":"f2b198e38823c9cccb029d6cb492e25230af34568ab56d46eafeaba9a4f2f9f5"} Jul 30 08:31:29.718975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:29.718924 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 30 08:31:29.718975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:29.718963 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 30 08:31:33.987448 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:33.987418 2576 generic.go:356] "Generic (PLEG): container finished" podID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerID="f2b198e38823c9cccb029d6cb492e25230af34568ab56d46eafeaba9a4f2f9f5" exitCode=0 Jul 30 08:31:33.987808 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:33.987500 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerDied","Data":"f2b198e38823c9cccb029d6cb492e25230af34568ab56d46eafeaba9a4f2f9f5"} Jul 30 08:31:34.996496 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:34.996455 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerStarted","Data":"1437b0c8195b8e406c0fc367d441dd3f7149e89a3f62ab4a369c21de4761dfc3"} Jul 30 08:31:34.997000 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:34.996518 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerStarted","Data":"7c348ab786b62b64e6f7d335b2fa87737475386e4dbd92d57930343362a28a80"} Jul 30 08:31:34.997000 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:34.996537 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerStarted","Data":"bc8d6418cede1eb4b647a6993413725985ef9fe0e6968442f3ef206ebe01fb97"} Jul 30 08:31:34.997000 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:34.996553 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerStarted","Data":"1a72bf8ade2507b15962ae130a5e138f1ce8b0e09c903735846fa159aedfc070"} Jul 30 08:31:35.020961 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:35.020892 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podStartSLOduration=13.02087269 podStartE2EDuration="13.02087269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:31:35.020008297 +0000 UTC m=+1189.279954658" watchObservedRunningTime="2026-07-30 08:31:35.02087269 +0000 UTC m=+1189.280819026" Jul 30 08:31:35.072557 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:35.072522 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:35.072849 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:35.072824 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:35.072849 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:35.072861 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:35.073051 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:35.072882 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:35.074930 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:35.074897 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Jul 30 08:31:35.075144 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:35.074923 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused" Jul 30 08:31:35.075327 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:35.074929 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 30 08:31:35.075455 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:35.074958 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 30 08:31:38.072774 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:38.072722 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused" Jul 30 08:31:38.072774 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:38.072754 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 30 08:31:38.072774 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:38.072751 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Jul 30 08:31:38.073357 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:38.072829 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 30 08:31:39.719255 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:39.719205 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 30 08:31:39.719877 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:39.719229 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 30 08:31:41.072667 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:41.072627 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:41.073115 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:41.072760 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:41.073409 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:41.073196 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:41.073409 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:41.073235 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:41.073409 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:41.073255 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:41.073409 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:41.073270 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:41.073826 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:41.073805 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:41.073939 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:41.073862 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:41.074547 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:41.074530 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:41.078078 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:41.078061 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:42.029329 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:42.029309 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:42.032253 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:42.032234 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:31:46.194287 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:46.194257 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:31:46.196255 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:46.196236 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:31:46.201056 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:46.201037 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:31:46.202416 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:46.202402 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:31:49.718985 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:49.718943 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 30 08:31:49.719431 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:49.719008 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 30 08:31:49.719431 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:49.719100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:31:49.719431 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:49.719165 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:31:52.071689 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.071657 2576 generic.go:356] "Generic (PLEG): container finished" podID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerID="6d1735d4841103cb609eff748ea133b74f3e7bacf12d73bc6c153ee8b02b0829" exitCode=137 Jul 30 08:31:52.071689 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.071683 2576 generic.go:356] "Generic (PLEG): container finished" podID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerID="8d1e379c9bdbdf0744104fc4387f6d4f977d28c75232e524369231352fae41f2" exitCode=137 Jul 30 08:31:52.072083 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.071726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerDied","Data":"6d1735d4841103cb609eff748ea133b74f3e7bacf12d73bc6c153ee8b02b0829"} Jul 30 08:31:52.072083 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.071772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerDied","Data":"8d1e379c9bdbdf0744104fc4387f6d4f977d28c75232e524369231352fae41f2"} Jul 30 08:31:52.542539 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.542518 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:31:52.622685 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.622659 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-feast-data\") pod \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " Jul 30 08:31:52.622840 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.622713 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-online-tls\") pod \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " Jul 30 08:31:52.622840 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.622747 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-registry-tls\") pod \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " Jul 30 08:31:52.622840 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.622771 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-offline-tls\") pod \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " Jul 30 08:31:52.622840 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.622800 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztbxl\" (UniqueName: \"kubernetes.io/projected/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-kube-api-access-ztbxl\") pod \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " Jul 30 08:31:52.622840 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.622835 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-ui-tls\") pod \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\" (UID: \"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba\") " Jul 30 08:31:52.623236 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.623206 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-feast-data" (OuterVolumeSpecName: "feast-data") pod "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" (UID: "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:31:52.625119 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.625087 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" (UID: "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:31:52.625271 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.625110 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-online-tls" (OuterVolumeSpecName: "online-tls") pod "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" (UID: "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:31:52.625271 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.625163 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-kube-api-access-ztbxl" (OuterVolumeSpecName: "kube-api-access-ztbxl") pod "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" (UID: "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba"). InnerVolumeSpecName "kube-api-access-ztbxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:31:52.625271 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.625206 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" (UID: "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:31:52.625271 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.625243 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" (UID: "30cdde88-0b6b-4f7e-bdaa-3b547c7352ba"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:31:52.723739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.723673 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:31:52.723739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.723704 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:31:52.723739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.723713 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-offline-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:31:52.723739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.723721 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztbxl\" (UniqueName: \"kubernetes.io/projected/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-kube-api-access-ztbxl\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:31:52.723739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.723731 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-ui-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:31:52.723739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:52.723740 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:31:53.078692 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:53.078658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" event={"ID":"30cdde88-0b6b-4f7e-bdaa-3b547c7352ba","Type":"ContainerDied","Data":"3be8946501e5f509591b85ae5a89a8d26b721684d4b58e59ea084919c44e76bb"} Jul 30 08:31:53.079138 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:53.078696 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz" Jul 30 08:31:53.079138 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:53.078718 2576 scope.go:117] "RemoveContainer" containerID="64d7dc83209f39324344ad500189efaa690e3d7466f5a601aabd504f691cc7b3" Jul 30 08:31:53.087239 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:53.087222 2576 scope.go:117] "RemoveContainer" containerID="6d1735d4841103cb609eff748ea133b74f3e7bacf12d73bc6c153ee8b02b0829" Jul 30 08:31:53.094056 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:53.094040 2576 scope.go:117] "RemoveContainer" containerID="714ff9961d79ee249be0aeb14214e708dfc0d327748a3ee0095070015be11e29" Jul 30 08:31:53.103120 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:53.102601 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz"] Jul 30 08:31:53.103926 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:53.103912 2576 scope.go:117] "RemoveContainer" containerID="8d1e379c9bdbdf0744104fc4387f6d4f977d28c75232e524369231352fae41f2" Jul 30 08:31:53.109118 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:53.109097 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-pbpdz"] Jul 30 08:31:53.111417 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:53.111395 2576 scope.go:117] "RemoveContainer" containerID="324d3e3afd94321a0538ab68a2e5d405784fbc0c11e61f93f627b562473fecac" Jul 30 08:31:53.117892 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:53.117877 2576 scope.go:117] "RemoveContainer" containerID="9b43a541ac9404e1ac4f686431a57188d437bafa2daff3429e26c9f2b6aa3f94" Jul 30 08:31:54.290547 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:31:54.290523 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" path="/var/lib/kubelet/pods/30cdde88-0b6b-4f7e-bdaa-3b547c7352ba/volumes" Jul 30 08:32:43.883878 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.883798 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v"] Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884102 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="offline" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884119 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="offline" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884140 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="feast-apply" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884146 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="feast-apply" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884153 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="feast-init" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884159 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="feast-init" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884164 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="ui" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884169 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="ui" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884177 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="online" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884182 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="online" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884192 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="registry" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884200 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="registry" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884243 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="offline" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884252 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="online" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884258 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="ui" Jul 30 08:32:43.884324 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.884265 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="30cdde88-0b6b-4f7e-bdaa-3b547c7352ba" containerName="registry" Jul 30 08:32:43.886396 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.886365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:43.889055 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.889036 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-2sf8m\"" Jul 30 08:32:43.889153 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.889103 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 30 08:32:43.889274 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.889261 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 30 08:32:43.889337 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.889321 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 30 08:32:43.889402 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.889352 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 30 08:32:43.889449 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.889411 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 30 08:32:43.889496 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.889461 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 30 08:32:43.899984 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:43.899963 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v"] Jul 30 08:32:44.079421 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.079391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/cfb2e695-6f9e-4144-a0ab-0277019050aa-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.079574 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.079431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb2e695-6f9e-4144-a0ab-0277019050aa-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.079574 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.079457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.079574 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.079482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6k62\" (UniqueName: \"kubernetes.io/projected/cfb2e695-6f9e-4144-a0ab-0277019050aa-kube-api-access-p6k62\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.079574 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.079528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.079574 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.079546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.180595 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.180522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.180595 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.180551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.180595 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.180589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/cfb2e695-6f9e-4144-a0ab-0277019050aa-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.180816 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.180629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb2e695-6f9e-4144-a0ab-0277019050aa-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.180816 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.180656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.180816 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.180692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6k62\" (UniqueName: \"kubernetes.io/projected/cfb2e695-6f9e-4144-a0ab-0277019050aa-kube-api-access-p6k62\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.181109 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.181087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb2e695-6f9e-4144-a0ab-0277019050aa-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.181411 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.181389 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/cfb2e695-6f9e-4144-a0ab-0277019050aa-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.182991 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.182970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.183100 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.183079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.183161 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.183147 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.189345 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.189323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6k62\" (UniqueName: \"kubernetes.io/projected/cfb2e695-6f9e-4144-a0ab-0277019050aa-kube-api-access-p6k62\") pod \"feast-simple-feast-remote-setup-db6955c6c-sgp7v\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.195065 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.195042 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:44.325346 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:44.325320 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v"] Jul 30 08:32:44.327173 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:32:44.327141 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfb2e695_6f9e_4144_a0ab_0277019050aa.slice/crio-9156e60c6527dc6c920a1fcd0e0c78ea96f3324def4a118d908f8075bac625a2 WatchSource:0}: Error finding container 9156e60c6527dc6c920a1fcd0e0c78ea96f3324def4a118d908f8075bac625a2: Status 404 returned error can't find the container with id 9156e60c6527dc6c920a1fcd0e0c78ea96f3324def4a118d908f8075bac625a2 Jul 30 08:32:45.260108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:45.260060 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerStarted","Data":"1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d"} Jul 30 08:32:45.260108 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:45.260109 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerStarted","Data":"9156e60c6527dc6c920a1fcd0e0c78ea96f3324def4a118d908f8075bac625a2"} Jul 30 08:32:49.279062 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:49.279030 2576 generic.go:356] "Generic (PLEG): container finished" podID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerID="1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d" exitCode=0 Jul 30 08:32:49.279502 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:49.279111 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerDied","Data":"1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d"} Jul 30 08:32:50.284388 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:50.284345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerStarted","Data":"430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e"} Jul 30 08:32:55.306349 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:55.305149 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerDied","Data":"430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e"} Jul 30 08:32:55.310976 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:55.307184 2576 generic.go:356] "Generic (PLEG): container finished" podID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerID="430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e" exitCode=0 Jul 30 08:32:56.315174 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:56.315132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerStarted","Data":"d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b"} Jul 30 08:32:56.315577 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:56.315197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerStarted","Data":"acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797"} Jul 30 08:32:56.315577 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:56.315217 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerStarted","Data":"5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365"} Jul 30 08:32:56.338687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:56.338637 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podStartSLOduration=13.33862181 podStartE2EDuration="13.33862181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:32:56.336317947 +0000 UTC m=+1270.596264322" watchObservedRunningTime="2026-07-30 08:32:56.33862181 +0000 UTC m=+1270.598568145" Jul 30 08:32:59.196075 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:59.196041 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:59.196552 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:59.196102 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:59.196552 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:59.196119 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:32:59.197970 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:59.197934 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Jul 30 08:32:59.198115 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:59.197973 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 30 08:32:59.198115 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:32:59.197946 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused" Jul 30 08:33:02.196667 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:33:02.196635 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:33:02.197121 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:33:02.196818 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:33:02.197218 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:33:02.197192 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:33:02.197272 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:33:02.197233 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:33:02.197525 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:33:02.197507 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:33:02.197579 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:33:02.197565 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:33:02.202958 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:33:02.202937 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:33:02.340678 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:33:02.340652 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:33:02.345010 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:33:02.344986 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:34:05.215290 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:05.214479 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v"] Jul 30 08:34:05.215290 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:05.214746 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="online" containerID="cri-o://5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365" gracePeriod=30 Jul 30 08:34:05.215290 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:05.215147 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="ui" containerID="cri-o://d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b" gracePeriod=30 Jul 30 08:34:05.216012 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:05.215325 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="offline" containerID="cri-o://acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797" gracePeriod=30 Jul 30 08:34:05.337300 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:05.337264 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp"] Jul 30 08:34:05.337849 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:05.337529 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="registry" containerID="cri-o://1a72bf8ade2507b15962ae130a5e138f1ce8b0e09c903735846fa159aedfc070" gracePeriod=30 Jul 30 08:34:05.337849 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:05.337643 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="online" containerID="cri-o://bc8d6418cede1eb4b647a6993413725985ef9fe0e6968442f3ef206ebe01fb97" gracePeriod=30 Jul 30 08:34:05.337849 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:05.337678 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="ui" containerID="cri-o://1437b0c8195b8e406c0fc367d441dd3f7149e89a3f62ab4a369c21de4761dfc3" gracePeriod=30 Jul 30 08:34:05.337849 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:05.337654 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="offline" containerID="cri-o://7c348ab786b62b64e6f7d335b2fa87737475386e4dbd92d57930343362a28a80" gracePeriod=30 Jul 30 08:34:06.573435 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:06.573388 2576 generic.go:356] "Generic (PLEG): container finished" podID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerID="d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b" exitCode=0 Jul 30 08:34:06.573435 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:06.573434 2576 generic.go:356] "Generic (PLEG): container finished" podID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerID="5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365" exitCode=0 Jul 30 08:34:06.573876 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:06.573395 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerDied","Data":"d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b"} Jul 30 08:34:06.573876 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:06.573527 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerDied","Data":"5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365"} Jul 30 08:34:06.576109 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:06.576086 2576 generic.go:356] "Generic (PLEG): container finished" podID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerID="1437b0c8195b8e406c0fc367d441dd3f7149e89a3f62ab4a369c21de4761dfc3" exitCode=0 Jul 30 08:34:06.576109 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:06.576104 2576 generic.go:356] "Generic (PLEG): container finished" podID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerID="bc8d6418cede1eb4b647a6993413725985ef9fe0e6968442f3ef206ebe01fb97" exitCode=0 Jul 30 08:34:06.576257 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:06.576131 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerDied","Data":"1437b0c8195b8e406c0fc367d441dd3f7149e89a3f62ab4a369c21de4761dfc3"} Jul 30 08:34:06.576257 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:06.576153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerDied","Data":"bc8d6418cede1eb4b647a6993413725985ef9fe0e6968442f3ef206ebe01fb97"} Jul 30 08:34:11.073289 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:11.073197 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 30 08:34:12.029624 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:12.029581 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 30 08:34:12.197962 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:12.197927 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Jul 30 08:34:12.341259 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:12.341232 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 30 08:34:21.073908 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:21.073843 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 30 08:34:22.030232 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:22.030194 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 30 08:34:22.197739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:22.197708 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Jul 30 08:34:22.340935 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:22.340910 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 30 08:34:31.073706 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:31.073644 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 30 08:34:31.074136 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:31.073813 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:34:32.030261 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:32.030227 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 30 08:34:32.030438 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:32.030327 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:34:32.197455 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:32.197418 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Jul 30 08:34:32.197857 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:32.197541 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:34:32.341068 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:32.341038 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 30 08:34:32.341229 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:32.341143 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:34:35.445927 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.445903 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:34:35.490182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.490114 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-offline-tls\") pod \"cfb2e695-6f9e-4144-a0ab-0277019050aa\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " Jul 30 08:34:35.490182 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.490161 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/cfb2e695-6f9e-4144-a0ab-0277019050aa-registry-tls\") pod \"cfb2e695-6f9e-4144-a0ab-0277019050aa\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " Jul 30 08:34:35.490321 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.490194 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb2e695-6f9e-4144-a0ab-0277019050aa-feast-data\") pod \"cfb2e695-6f9e-4144-a0ab-0277019050aa\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " Jul 30 08:34:35.490321 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.490304 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-online-tls\") pod \"cfb2e695-6f9e-4144-a0ab-0277019050aa\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " Jul 30 08:34:35.490426 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.490403 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6k62\" (UniqueName: \"kubernetes.io/projected/cfb2e695-6f9e-4144-a0ab-0277019050aa-kube-api-access-p6k62\") pod \"cfb2e695-6f9e-4144-a0ab-0277019050aa\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " Jul 30 08:34:35.490476 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.490449 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-ui-tls\") pod \"cfb2e695-6f9e-4144-a0ab-0277019050aa\" (UID: \"cfb2e695-6f9e-4144-a0ab-0277019050aa\") " Jul 30 08:34:35.490574 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.490550 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb2e695-6f9e-4144-a0ab-0277019050aa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cfb2e695-6f9e-4144-a0ab-0277019050aa" (UID: "cfb2e695-6f9e-4144-a0ab-0277019050aa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 08:34:35.490682 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.490635 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb2e695-6f9e-4144-a0ab-0277019050aa-feast-data" (OuterVolumeSpecName: "feast-data") pod "cfb2e695-6f9e-4144-a0ab-0277019050aa" (UID: "cfb2e695-6f9e-4144-a0ab-0277019050aa"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:34:35.490775 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.490754 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb2e695-6f9e-4144-a0ab-0277019050aa-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:35.490775 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.490777 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/cfb2e695-6f9e-4144-a0ab-0277019050aa-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:35.492431 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.492390 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb2e695-6f9e-4144-a0ab-0277019050aa-kube-api-access-p6k62" (OuterVolumeSpecName: "kube-api-access-p6k62") pod "cfb2e695-6f9e-4144-a0ab-0277019050aa" (UID: "cfb2e695-6f9e-4144-a0ab-0277019050aa"). InnerVolumeSpecName "kube-api-access-p6k62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:34:35.492519 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.492478 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "cfb2e695-6f9e-4144-a0ab-0277019050aa" (UID: "cfb2e695-6f9e-4144-a0ab-0277019050aa"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:34:35.492562 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.492543 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-online-tls" (OuterVolumeSpecName: "online-tls") pod "cfb2e695-6f9e-4144-a0ab-0277019050aa" (UID: "cfb2e695-6f9e-4144-a0ab-0277019050aa"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:34:35.492597 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.492563 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "cfb2e695-6f9e-4144-a0ab-0277019050aa" (UID: "cfb2e695-6f9e-4144-a0ab-0277019050aa"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:34:35.591889 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.591866 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6k62\" (UniqueName: \"kubernetes.io/projected/cfb2e695-6f9e-4144-a0ab-0277019050aa-kube-api-access-p6k62\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:35.591889 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.591887 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-ui-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:35.592023 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.591896 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-offline-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:35.592023 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.591906 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cfb2e695-6f9e-4144-a0ab-0277019050aa-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:35.681009 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.680983 2576 generic.go:356] "Generic (PLEG): container finished" podID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerID="7c348ab786b62b64e6f7d335b2fa87737475386e4dbd92d57930343362a28a80" exitCode=137 Jul 30 08:34:35.681009 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.681003 2576 generic.go:356] "Generic (PLEG): container finished" podID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerID="1a72bf8ade2507b15962ae130a5e138f1ce8b0e09c903735846fa159aedfc070" exitCode=137 Jul 30 08:34:35.681172 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.681053 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerDied","Data":"7c348ab786b62b64e6f7d335b2fa87737475386e4dbd92d57930343362a28a80"} Jul 30 08:34:35.681172 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.681096 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerDied","Data":"1a72bf8ade2507b15962ae130a5e138f1ce8b0e09c903735846fa159aedfc070"} Jul 30 08:34:35.683345 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.683324 2576 generic.go:356] "Generic (PLEG): container finished" podID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerID="acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797" exitCode=137 Jul 30 08:34:35.683433 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.683396 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerDied","Data":"acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797"} Jul 30 08:34:35.683433 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.683427 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" event={"ID":"cfb2e695-6f9e-4144-a0ab-0277019050aa","Type":"ContainerDied","Data":"9156e60c6527dc6c920a1fcd0e0c78ea96f3324def4a118d908f8075bac625a2"} Jul 30 08:34:35.683529 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.683446 2576 scope.go:117] "RemoveContainer" containerID="d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b" Jul 30 08:34:35.683529 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.683402 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v" Jul 30 08:34:35.691809 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.691796 2576 scope.go:117] "RemoveContainer" containerID="acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797" Jul 30 08:34:35.698798 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.698772 2576 scope.go:117] "RemoveContainer" containerID="5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365" Jul 30 08:34:35.704694 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.704675 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v"] Jul 30 08:34:35.705928 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.705917 2576 scope.go:117] "RemoveContainer" containerID="430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e" Jul 30 08:34:35.710148 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.710127 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-sgp7v"] Jul 30 08:34:35.712897 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.712882 2576 scope.go:117] "RemoveContainer" containerID="1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d" Jul 30 08:34:35.719411 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.719395 2576 scope.go:117] "RemoveContainer" containerID="d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b" Jul 30 08:34:35.719655 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:34:35.719636 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b\": container with ID starting with d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b not found: ID does not exist" containerID="d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b" Jul 30 08:34:35.719719 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.719663 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b"} err="failed to get container status \"d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b\": rpc error: code = NotFound desc = could not find container \"d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b\": container with ID starting with d7c38d8ee1a755ea7df985eed5d49da5910596ba0074d4e9b7b0e97d6876855b not found: ID does not exist" Jul 30 08:34:35.719719 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.719686 2576 scope.go:117] "RemoveContainer" containerID="acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797" Jul 30 08:34:35.719924 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:34:35.719907 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797\": container with ID starting with acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797 not found: ID does not exist" containerID="acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797" Jul 30 08:34:35.719970 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.719928 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797"} err="failed to get container status \"acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797\": rpc error: code = NotFound desc = could not find container \"acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797\": container with ID starting with acdcd0fb882e8651f1b63b72d5919efc63de4b13bae926b0afe768f24c1d5797 not found: ID does not exist" Jul 30 08:34:35.719970 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.719943 2576 scope.go:117] "RemoveContainer" containerID="5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365" Jul 30 08:34:35.720151 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:34:35.720138 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365\": container with ID starting with 5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365 not found: ID does not exist" containerID="5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365" Jul 30 08:34:35.720191 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.720154 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365"} err="failed to get container status \"5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365\": rpc error: code = NotFound desc = could not find container \"5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365\": container with ID starting with 5c6109960cfa812130b5e2afdf928baacb406c2c122f1f579f34899ebf99c365 not found: ID does not exist" Jul 30 08:34:35.720191 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.720165 2576 scope.go:117] "RemoveContainer" containerID="430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e" Jul 30 08:34:35.720348 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:34:35.720332 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e\": container with ID starting with 430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e not found: ID does not exist" containerID="430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e" Jul 30 08:34:35.720416 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.720351 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e"} err="failed to get container status \"430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e\": rpc error: code = NotFound desc = could not find container \"430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e\": container with ID starting with 430f6b060dc1de7f4c29d0e09b1925313532aebb3a0dfd0f5cb58a17219fd10e not found: ID does not exist" Jul 30 08:34:35.720416 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.720364 2576 scope.go:117] "RemoveContainer" containerID="1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d" Jul 30 08:34:35.720589 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:34:35.720573 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d\": container with ID starting with 1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d not found: ID does not exist" containerID="1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d" Jul 30 08:34:35.720626 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.720593 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d"} err="failed to get container status \"1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d\": rpc error: code = NotFound desc = could not find container \"1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d\": container with ID starting with 1a9889dade40033b2f4814d71d47ad07e9f949daa0495faf412316134506453d not found: ID does not exist" Jul 30 08:34:35.978178 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:35.978153 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:34:36.095195 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.095171 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qjrj\" (UniqueName: \"kubernetes.io/projected/9f60a33c-ff9d-400a-a75c-88ddde4fb480-kube-api-access-9qjrj\") pod \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " Jul 30 08:34:36.095317 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.095208 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-ui-tls\") pod \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " Jul 30 08:34:36.095317 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.095278 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-registry-tls\") pod \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " Jul 30 08:34:36.095470 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.095317 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-offline-tls\") pod \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " Jul 30 08:34:36.095470 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.095344 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9f60a33c-ff9d-400a-a75c-88ddde4fb480-feast-data\") pod \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " Jul 30 08:34:36.095470 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.095369 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-online-tls\") pod \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\" (UID: \"9f60a33c-ff9d-400a-a75c-88ddde4fb480\") " Jul 30 08:34:36.095920 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.095877 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f60a33c-ff9d-400a-a75c-88ddde4fb480-feast-data" (OuterVolumeSpecName: "feast-data") pod "9f60a33c-ff9d-400a-a75c-88ddde4fb480" (UID: "9f60a33c-ff9d-400a-a75c-88ddde4fb480"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:34:36.097242 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.097219 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f60a33c-ff9d-400a-a75c-88ddde4fb480-kube-api-access-9qjrj" (OuterVolumeSpecName: "kube-api-access-9qjrj") pod "9f60a33c-ff9d-400a-a75c-88ddde4fb480" (UID: "9f60a33c-ff9d-400a-a75c-88ddde4fb480"). InnerVolumeSpecName "kube-api-access-9qjrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:34:36.097242 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.097234 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "9f60a33c-ff9d-400a-a75c-88ddde4fb480" (UID: "9f60a33c-ff9d-400a-a75c-88ddde4fb480"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:34:36.097365 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.097216 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9f60a33c-ff9d-400a-a75c-88ddde4fb480" (UID: "9f60a33c-ff9d-400a-a75c-88ddde4fb480"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:34:36.097662 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.097644 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-online-tls" (OuterVolumeSpecName: "online-tls") pod "9f60a33c-ff9d-400a-a75c-88ddde4fb480" (UID: "9f60a33c-ff9d-400a-a75c-88ddde4fb480"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:34:36.097711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.097668 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "9f60a33c-ff9d-400a-a75c-88ddde4fb480" (UID: "9f60a33c-ff9d-400a-a75c-88ddde4fb480"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:34:36.196241 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.196193 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:36.196241 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.196211 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-offline-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:36.196241 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.196220 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9f60a33c-ff9d-400a-a75c-88ddde4fb480-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:36.196241 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.196227 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:36.196241 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.196235 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qjrj\" (UniqueName: \"kubernetes.io/projected/9f60a33c-ff9d-400a-a75c-88ddde4fb480-kube-api-access-9qjrj\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:36.196451 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.196244 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9f60a33c-ff9d-400a-a75c-88ddde4fb480-ui-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:34:36.290338 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.290318 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" path="/var/lib/kubelet/pods/cfb2e695-6f9e-4144-a0ab-0277019050aa/volumes" Jul 30 08:34:36.690916 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.690886 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" event={"ID":"9f60a33c-ff9d-400a-a75c-88ddde4fb480","Type":"ContainerDied","Data":"627e1fcd7f2e25e0a79f160963d9698c6adcc5435d27398530f72d3b5c6609b7"} Jul 30 08:34:36.691218 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.690935 2576 scope.go:117] "RemoveContainer" containerID="1437b0c8195b8e406c0fc367d441dd3f7149e89a3f62ab4a369c21de4761dfc3" Jul 30 08:34:36.691218 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.690935 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp" Jul 30 08:34:36.698635 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.698620 2576 scope.go:117] "RemoveContainer" containerID="7c348ab786b62b64e6f7d335b2fa87737475386e4dbd92d57930343362a28a80" Jul 30 08:34:36.705699 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.705682 2576 scope.go:117] "RemoveContainer" containerID="bc8d6418cede1eb4b647a6993413725985ef9fe0e6968442f3ef206ebe01fb97" Jul 30 08:34:36.709680 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.709660 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp"] Jul 30 08:34:36.713696 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.713679 2576 scope.go:117] "RemoveContainer" containerID="1a72bf8ade2507b15962ae130a5e138f1ce8b0e09c903735846fa159aedfc070" Jul 30 08:34:36.713925 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.713906 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wpgrp"] Jul 30 08:34:36.720275 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.720261 2576 scope.go:117] "RemoveContainer" containerID="f2b198e38823c9cccb029d6cb492e25230af34568ab56d46eafeaba9a4f2f9f5" Jul 30 08:34:36.729435 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:36.729421 2576 scope.go:117] "RemoveContainer" containerID="72b25d770ba303516e7cafcff2460ae48a3eb45825d785be4533693a0d12342d" Jul 30 08:34:38.290514 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:34:38.290483 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" path="/var/lib/kubelet/pods/9f60a33c-ff9d-400a-a75c-88ddde4fb480/volumes" Jul 30 08:35:15.459314 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459244 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt"] Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459512 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="ui" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459524 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="ui" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459536 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="registry" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459541 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="registry" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459547 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="online" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459553 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="online" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459559 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="online" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459563 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="online" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459572 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="ui" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459577 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="ui" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459586 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="offline" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459591 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="offline" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459597 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="feast-init" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459602 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="feast-init" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459608 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="feast-apply" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459612 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="feast-apply" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459619 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="feast-init" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459623 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="feast-init" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459630 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="feast-apply" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459634 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="feast-apply" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459641 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="offline" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459646 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="offline" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459685 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="online" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459693 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="registry" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459698 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="offline" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459703 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="ui" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459710 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="online" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459715 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb2e695-6f9e-4144-a0ab-0277019050aa" containerName="offline" Jul 30 08:35:15.459739 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.459720 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f60a33c-ff9d-400a-a75c-88ddde4fb480" containerName="ui" Jul 30 08:35:15.464623 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.464581 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.467355 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.467330 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 30 08:35:15.467501 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.467413 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 30 08:35:15.467761 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.467742 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 30 08:35:15.468432 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.468416 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-h292j\"" Jul 30 08:35:15.468526 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.468508 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 30 08:35:15.468581 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.468536 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 30 08:35:15.468581 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.468537 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 30 08:35:15.475616 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.475593 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt"] Jul 30 08:35:15.573621 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.573591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2lpz\" (UniqueName: \"kubernetes.io/projected/513e4049-c76b-4f14-9fe8-62323af1fbe8-kube-api-access-m2lpz\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.573621 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.573623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.573817 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.573646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.573817 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.573661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.573817 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.573697 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.573817 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.573760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/513e4049-c76b-4f14-9fe8-62323af1fbe8-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.674148 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.674121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.674266 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.674156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/513e4049-c76b-4f14-9fe8-62323af1fbe8-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.674266 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.674189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2lpz\" (UniqueName: \"kubernetes.io/projected/513e4049-c76b-4f14-9fe8-62323af1fbe8-kube-api-access-m2lpz\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.674346 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.674301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.674425 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.674349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.674425 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.674392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.674582 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.674565 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/513e4049-c76b-4f14-9fe8-62323af1fbe8-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.676644 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.676613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.676742 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.676712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.676742 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.676727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.676821 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.676780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.682068 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.682048 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2lpz\" (UniqueName: \"kubernetes.io/projected/513e4049-c76b-4f14-9fe8-62323af1fbe8-kube-api-access-m2lpz\") pod \"feast-simple-feast-setup-777d6cc998-qbsrt\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.785955 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.785901 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:15.912982 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:15.912959 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt"] Jul 30 08:35:15.915258 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:35:15.915233 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod513e4049_c76b_4f14_9fe8_62323af1fbe8.slice/crio-1cf4925a4e0cd4e38921c07995c26e0b2bd13030d58083ed16567c22ce48c2d4 WatchSource:0}: Error finding container 1cf4925a4e0cd4e38921c07995c26e0b2bd13030d58083ed16567c22ce48c2d4: Status 404 returned error can't find the container with id 1cf4925a4e0cd4e38921c07995c26e0b2bd13030d58083ed16567c22ce48c2d4 Jul 30 08:35:16.830329 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:16.830295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerStarted","Data":"d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3"} Jul 30 08:35:16.830699 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:16.830337 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerStarted","Data":"1cf4925a4e0cd4e38921c07995c26e0b2bd13030d58083ed16567c22ce48c2d4"} Jul 30 08:35:20.856444 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:20.856410 2576 generic.go:356] "Generic (PLEG): container finished" podID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerID="d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3" exitCode=0 Jul 30 08:35:20.856755 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:20.856447 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerDied","Data":"d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3"} Jul 30 08:35:21.862268 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:21.862237 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerStarted","Data":"26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046"} Jul 30 08:35:26.880315 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:26.880285 2576 generic.go:356] "Generic (PLEG): container finished" podID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerID="26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046" exitCode=0 Jul 30 08:35:26.880702 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:26.880323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerDied","Data":"26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046"} Jul 30 08:35:27.889453 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:27.889408 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerStarted","Data":"e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc"} Jul 30 08:35:27.889911 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:27.889471 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerStarted","Data":"dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e"} Jul 30 08:35:27.889911 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:27.889493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerStarted","Data":"ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0"} Jul 30 08:35:27.889911 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:27.889510 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerStarted","Data":"81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7"} Jul 30 08:35:27.915133 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:27.915036 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podStartSLOduration=12.915015194 podStartE2EDuration="12.915015194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:35:27.912215294 +0000 UTC m=+1422.172161633" watchObservedRunningTime="2026-07-30 08:35:27.915015194 +0000 UTC m=+1422.174961531" Jul 30 08:35:30.786470 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:30.786435 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:30.786938 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:30.786491 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:30.786938 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:30.786510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:30.786938 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:30.786527 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:30.788593 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:30.788532 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused" Jul 30 08:35:30.789037 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:30.788561 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 30 08:35:30.789037 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:30.788568 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Jul 30 08:35:30.789037 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:30.788600 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Jul 30 08:35:33.787669 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.787635 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:33.788058 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.787786 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:33.788058 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.787941 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:33.788391 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.788285 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:33.788391 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.788328 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:33.788391 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.788346 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:33.788595 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.788555 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:33.788756 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.788738 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:33.789495 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.789482 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:33.792795 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.792777 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:33.917222 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.917201 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:35:33.919946 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:35:33.919931 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:36:36.565572 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.565535 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt"] Jul 30 08:36:36.566140 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.565840 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="registry" containerID="cri-o://81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7" gracePeriod=30 Jul 30 08:36:36.566140 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.565897 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="offline" containerID="cri-o://dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e" gracePeriod=30 Jul 30 08:36:36.566140 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.565922 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="online" containerID="cri-o://ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0" gracePeriod=30 Jul 30 08:36:36.566140 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.566027 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="ui" containerID="cri-o://e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc" gracePeriod=30 Jul 30 08:36:36.833062 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.832981 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv"] Jul 30 08:36:36.836654 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.836634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:36.838924 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.838897 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-zc2p9\"" Jul 30 08:36:36.850527 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.850501 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv"] Jul 30 08:36:36.926921 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.926885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:36.927144 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.926935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s526x\" (UniqueName: \"kubernetes.io/projected/7401f3b1-c49d-419e-99b9-ce916679026f-kube-api-access-s526x\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:36.927144 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.927019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:36.927144 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.927044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:36.927144 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.927068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:36.927144 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:36.927085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7401f3b1-c49d-419e-99b9-ce916679026f-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.027808 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.027778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.027904 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.027814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.027904 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.027830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.027904 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.027846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7401f3b1-c49d-419e-99b9-ce916679026f-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.027904 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.027867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.028068 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.027939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s526x\" (UniqueName: \"kubernetes.io/projected/7401f3b1-c49d-419e-99b9-ce916679026f-kube-api-access-s526x\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.028068 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:36:37.027960 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 30 08:36:37.028068 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:36:37.028001 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 30 08:36:37.028068 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:36:37.028024 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-online-tls podName:7401f3b1-c49d-419e-99b9-ce916679026f nodeName:}" failed. No retries permitted until 2026-07-30 08:36:37.528003716 +0000 UTC m=+1491.787950033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-online-tls") pod "feast-simple-feast-setup-777d6cc998-ppkdv" (UID: "7401f3b1-c49d-419e-99b9-ce916679026f") : secret "feast-simple-feast-setup-online-tls" not found Jul 30 08:36:37.028068 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:36:37.027960 2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:36:37.028329 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:36:37.028075 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-registry-tls podName:7401f3b1-c49d-419e-99b9-ce916679026f nodeName:}" failed. No retries permitted until 2026-07-30 08:36:37.528056735 +0000 UTC m=+1491.788003068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-registry-tls") pod "feast-simple-feast-setup-777d6cc998-ppkdv" (UID: "7401f3b1-c49d-419e-99b9-ce916679026f") : secret "feast-simple-feast-setup-registry-tls" not found Jul 30 08:36:37.028329 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:36:37.028094 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-offline-tls podName:7401f3b1-c49d-419e-99b9-ce916679026f nodeName:}" failed. No retries permitted until 2026-07-30 08:36:37.528085133 +0000 UTC m=+1491.788031452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-offline-tls") pod "feast-simple-feast-setup-777d6cc998-ppkdv" (UID: "7401f3b1-c49d-419e-99b9-ce916679026f") : secret "feast-simple-feast-setup-offline-tls" not found Jul 30 08:36:37.028329 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.028163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7401f3b1-c49d-419e-99b9-ce916679026f-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.030340 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.030321 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.038052 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.038030 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s526x\" (UniqueName: \"kubernetes.io/projected/7401f3b1-c49d-419e-99b9-ce916679026f-kube-api-access-s526x\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.149796 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.149717 2576 generic.go:356] "Generic (PLEG): container finished" podID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerID="e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc" exitCode=0 Jul 30 08:36:37.149796 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.149770 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerDied","Data":"e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc"} Jul 30 08:36:37.532569 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.532508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.532569 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.532538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.532569 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.532556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.534767 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.534740 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.534863 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.534810 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.534910 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.534889 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppkdv\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.747768 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.747749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:37.876884 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:37.876844 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv"] Jul 30 08:36:37.879011 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:36:37.878988 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7401f3b1_c49d_419e_99b9_ce916679026f.slice/crio-2e57c2e9bfcd59e1b8c7eb278f822de61c3beb3445f06bff3b8f8beb29d5f9fa WatchSource:0}: Error finding container 2e57c2e9bfcd59e1b8c7eb278f822de61c3beb3445f06bff3b8f8beb29d5f9fa: Status 404 returned error can't find the container with id 2e57c2e9bfcd59e1b8c7eb278f822de61c3beb3445f06bff3b8f8beb29d5f9fa Jul 30 08:36:38.158351 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:38.158259 2576 generic.go:356] "Generic (PLEG): container finished" podID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerID="ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0" exitCode=0 Jul 30 08:36:38.158534 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:38.158365 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerDied","Data":"ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0"} Jul 30 08:36:38.159956 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:38.159925 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerStarted","Data":"5f0073d940bde409bf5c8799c27e3e96e7ffb4615f8028bcd421530508f2979c"} Jul 30 08:36:38.160071 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:38.159969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerStarted","Data":"2e57c2e9bfcd59e1b8c7eb278f822de61c3beb3445f06bff3b8f8beb29d5f9fa"} Jul 30 08:36:43.181254 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:43.181228 2576 generic.go:356] "Generic (PLEG): container finished" podID="7401f3b1-c49d-419e-99b9-ce916679026f" containerID="5f0073d940bde409bf5c8799c27e3e96e7ffb4615f8028bcd421530508f2979c" exitCode=0 Jul 30 08:36:43.181629 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:43.181268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerDied","Data":"5f0073d940bde409bf5c8799c27e3e96e7ffb4615f8028bcd421530508f2979c"} Jul 30 08:36:43.788788 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:43.788746 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Jul 30 08:36:43.917601 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:43.917563 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 30 08:36:44.193468 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:44.193386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerStarted","Data":"bac68a7df43563f1a2b8aa77d2fe23c36c2421761dd8d2d5945ea185f6949548"} Jul 30 08:36:46.219579 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:46.219550 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:36:46.220033 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:46.219762 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:36:46.226199 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:46.226180 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:36:46.226608 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:46.226590 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:36:49.213056 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:49.213022 2576 generic.go:356] "Generic (PLEG): container finished" podID="7401f3b1-c49d-419e-99b9-ce916679026f" containerID="bac68a7df43563f1a2b8aa77d2fe23c36c2421761dd8d2d5945ea185f6949548" exitCode=0 Jul 30 08:36:49.213406 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:49.213080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerDied","Data":"bac68a7df43563f1a2b8aa77d2fe23c36c2421761dd8d2d5945ea185f6949548"} Jul 30 08:36:50.222992 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:50.222940 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerStarted","Data":"daf7cfc93e7e8e4084344d933bc1e4a7c1bfb7ba2bdc31f3882f8d26c3744c49"} Jul 30 08:36:50.223471 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:50.223000 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerStarted","Data":"f97522d1efdd194930bbb1e67d0f1e6574f1f14382ec6caabea161cca889f38c"} Jul 30 08:36:50.223471 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:50.223022 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerStarted","Data":"b73acd50bf02f2c7436aa317cabb014d3c343f2d48a312c362e298fe9ff757f2"} Jul 30 08:36:50.223471 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:50.223040 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerStarted","Data":"806dc42fd81578d8ebf9772242816bcdd19f53bce16a073aeefeefd199201a80"} Jul 30 08:36:50.268892 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:50.268832 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podStartSLOduration=14.268812097 podStartE2EDuration="14.268812097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:36:50.268692809 +0000 UTC m=+1504.528639145" watchObservedRunningTime="2026-07-30 08:36:50.268812097 +0000 UTC m=+1504.528758436" Jul 30 08:36:52.748777 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:52.748729 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:52.749343 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:52.749311 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:52.749343 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:52.749344 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:52.749551 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:52.749362 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:52.751021 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:52.750988 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Jul 30 08:36:52.751138 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:52.751037 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Jul 30 08:36:52.751138 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:52.751000 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused" Jul 30 08:36:52.751328 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:52.751295 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 30 08:36:53.788957 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:53.788909 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Jul 30 08:36:53.918464 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:53.918401 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 30 08:36:55.749090 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:55.749058 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:55.749521 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:55.749301 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:55.749692 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:55.749669 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:55.749807 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:55.749758 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:55.749949 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:55.749933 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:55.750025 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:55.749996 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:55.750121 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:55.750052 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:55.754455 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:55.754438 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:56.255779 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:56.255757 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:56.255888 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:56.255790 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:56.256259 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:56.256244 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:36:56.258729 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:36:56.258714 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:37:03.788454 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:03.788415 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Jul 30 08:37:03.788905 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:03.788551 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:37:03.917292 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:03.917260 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 30 08:37:03.917439 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:03.917357 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:37:07.204899 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.204878 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:37:07.299523 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.299492 2576 generic.go:356] "Generic (PLEG): container finished" podID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerID="dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e" exitCode=137 Jul 30 08:37:07.299523 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.299517 2576 generic.go:356] "Generic (PLEG): container finished" podID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerID="81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7" exitCode=137 Jul 30 08:37:07.299672 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.299534 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerDied","Data":"dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e"} Jul 30 08:37:07.299672 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.299553 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" Jul 30 08:37:07.299672 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.299577 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerDied","Data":"81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7"} Jul 30 08:37:07.299672 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.299592 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt" event={"ID":"513e4049-c76b-4f14-9fe8-62323af1fbe8","Type":"ContainerDied","Data":"1cf4925a4e0cd4e38921c07995c26e0b2bd13030d58083ed16567c22ce48c2d4"} Jul 30 08:37:07.299672 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.299611 2576 scope.go:117] "RemoveContainer" containerID="e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc" Jul 30 08:37:07.307749 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.307731 2576 scope.go:117] "RemoveContainer" containerID="dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e" Jul 30 08:37:07.314653 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.314640 2576 scope.go:117] "RemoveContainer" containerID="ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0" Jul 30 08:37:07.321272 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.321256 2576 scope.go:117] "RemoveContainer" containerID="81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7" Jul 30 08:37:07.328087 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.328069 2576 scope.go:117] "RemoveContainer" containerID="26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046" Jul 30 08:37:07.334359 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.334344 2576 scope.go:117] "RemoveContainer" containerID="d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3" Jul 30 08:37:07.340546 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.340530 2576 scope.go:117] "RemoveContainer" containerID="e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc" Jul 30 08:37:07.340799 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:37:07.340775 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc\": container with ID starting with e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc not found: ID does not exist" containerID="e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc" Jul 30 08:37:07.340847 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.340801 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc"} err="failed to get container status \"e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc\": rpc error: code = NotFound desc = could not find container \"e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc\": container with ID starting with e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc not found: ID does not exist" Jul 30 08:37:07.340847 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.340816 2576 scope.go:117] "RemoveContainer" containerID="dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e" Jul 30 08:37:07.341053 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:37:07.341039 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e\": container with ID starting with dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e not found: ID does not exist" containerID="dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e" Jul 30 08:37:07.341101 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.341059 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e"} err="failed to get container status \"dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e\": rpc error: code = NotFound desc = could not find container \"dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e\": container with ID starting with dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e not found: ID does not exist" Jul 30 08:37:07.341101 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.341083 2576 scope.go:117] "RemoveContainer" containerID="ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0" Jul 30 08:37:07.341330 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:37:07.341310 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0\": container with ID starting with ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0 not found: ID does not exist" containerID="ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0" Jul 30 08:37:07.341388 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.341337 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0"} err="failed to get container status \"ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0\": rpc error: code = NotFound desc = could not find container \"ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0\": container with ID starting with ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0 not found: ID does not exist" Jul 30 08:37:07.341388 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.341353 2576 scope.go:117] "RemoveContainer" containerID="81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7" Jul 30 08:37:07.341595 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:37:07.341579 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7\": container with ID starting with 81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7 not found: ID does not exist" containerID="81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7" Jul 30 08:37:07.341651 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.341598 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7"} err="failed to get container status \"81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7\": rpc error: code = NotFound desc = could not find container \"81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7\": container with ID starting with 81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7 not found: ID does not exist" Jul 30 08:37:07.341651 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.341621 2576 scope.go:117] "RemoveContainer" containerID="26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046" Jul 30 08:37:07.341838 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:37:07.341823 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046\": container with ID starting with 26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046 not found: ID does not exist" containerID="26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046" Jul 30 08:37:07.341895 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.341840 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046"} err="failed to get container status \"26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046\": rpc error: code = NotFound desc = could not find container \"26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046\": container with ID starting with 26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046 not found: ID does not exist" Jul 30 08:37:07.341895 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.341854 2576 scope.go:117] "RemoveContainer" containerID="d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3" Jul 30 08:37:07.342078 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:37:07.342062 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3\": container with ID starting with d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3 not found: ID does not exist" containerID="d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3" Jul 30 08:37:07.342118 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.342082 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3"} err="failed to get container status \"d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3\": rpc error: code = NotFound desc = could not find container \"d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3\": container with ID starting with d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3 not found: ID does not exist" Jul 30 08:37:07.342118 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.342094 2576 scope.go:117] "RemoveContainer" containerID="e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc" Jul 30 08:37:07.342292 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.342273 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc"} err="failed to get container status \"e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc\": rpc error: code = NotFound desc = could not find container \"e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc\": container with ID starting with e19ab394fce1de6f09cb2bd39d3e5f6528215bc0ce0eae2c0faa59f8737ff9cc not found: ID does not exist" Jul 30 08:37:07.342334 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.342293 2576 scope.go:117] "RemoveContainer" containerID="dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e" Jul 30 08:37:07.342499 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.342484 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e"} err="failed to get container status \"dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e\": rpc error: code = NotFound desc = could not find container \"dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e\": container with ID starting with dc988858b85e4ef33706fb7fca54bd63709eb92739d85d594e90d03a40fa2c4e not found: ID does not exist" Jul 30 08:37:07.342547 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.342500 2576 scope.go:117] "RemoveContainer" containerID="ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0" Jul 30 08:37:07.342702 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.342681 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0"} err="failed to get container status \"ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0\": rpc error: code = NotFound desc = could not find container \"ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0\": container with ID starting with ca31b0b4dd76a4c685b1753782e56954f07a2b736b677dbe6634ee79622822c0 not found: ID does not exist" Jul 30 08:37:07.342776 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.342703 2576 scope.go:117] "RemoveContainer" containerID="81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7" Jul 30 08:37:07.342900 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.342884 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7"} err="failed to get container status \"81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7\": rpc error: code = NotFound desc = could not find container \"81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7\": container with ID starting with 81262e8bcb81f09f490e997e519a9bfeb61ca93176b08bbebd8283536860a0c7 not found: ID does not exist" Jul 30 08:37:07.342973 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.342901 2576 scope.go:117] "RemoveContainer" containerID="26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046" Jul 30 08:37:07.343115 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.343097 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046"} err="failed to get container status \"26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046\": rpc error: code = NotFound desc = could not find container \"26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046\": container with ID starting with 26e99620d3f64092a9050faf6f0cc9f40f61692b6884a3eca7d6a819d569d046 not found: ID does not exist" Jul 30 08:37:07.343160 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.343116 2576 scope.go:117] "RemoveContainer" containerID="d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3" Jul 30 08:37:07.343295 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.343278 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3"} err="failed to get container status \"d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3\": rpc error: code = NotFound desc = could not find container \"d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3\": container with ID starting with d2c27a01cfaabd54f20fefd37a7b78a0e29e30cb1ac74afa81471c0ecab70ec3 not found: ID does not exist" Jul 30 08:37:07.359610 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.359592 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-online-tls\") pod \"513e4049-c76b-4f14-9fe8-62323af1fbe8\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " Jul 30 08:37:07.359684 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.359614 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-offline-tls\") pod \"513e4049-c76b-4f14-9fe8-62323af1fbe8\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " Jul 30 08:37:07.359684 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.359644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/513e4049-c76b-4f14-9fe8-62323af1fbe8-feast-data\") pod \"513e4049-c76b-4f14-9fe8-62323af1fbe8\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " Jul 30 08:37:07.359775 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.359683 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2lpz\" (UniqueName: \"kubernetes.io/projected/513e4049-c76b-4f14-9fe8-62323af1fbe8-kube-api-access-m2lpz\") pod \"513e4049-c76b-4f14-9fe8-62323af1fbe8\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " Jul 30 08:37:07.359775 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.359708 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-ui-tls\") pod \"513e4049-c76b-4f14-9fe8-62323af1fbe8\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " Jul 30 08:37:07.359775 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.359735 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-registry-tls\") pod \"513e4049-c76b-4f14-9fe8-62323af1fbe8\" (UID: \"513e4049-c76b-4f14-9fe8-62323af1fbe8\") " Jul 30 08:37:07.360207 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.360182 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513e4049-c76b-4f14-9fe8-62323af1fbe8-feast-data" (OuterVolumeSpecName: "feast-data") pod "513e4049-c76b-4f14-9fe8-62323af1fbe8" (UID: "513e4049-c76b-4f14-9fe8-62323af1fbe8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:37:07.361755 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.361727 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513e4049-c76b-4f14-9fe8-62323af1fbe8-kube-api-access-m2lpz" (OuterVolumeSpecName: "kube-api-access-m2lpz") pod "513e4049-c76b-4f14-9fe8-62323af1fbe8" (UID: "513e4049-c76b-4f14-9fe8-62323af1fbe8"). InnerVolumeSpecName "kube-api-access-m2lpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:37:07.361850 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.361748 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "513e4049-c76b-4f14-9fe8-62323af1fbe8" (UID: "513e4049-c76b-4f14-9fe8-62323af1fbe8"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:37:07.361902 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.361866 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-online-tls" (OuterVolumeSpecName: "online-tls") pod "513e4049-c76b-4f14-9fe8-62323af1fbe8" (UID: "513e4049-c76b-4f14-9fe8-62323af1fbe8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:37:07.361950 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.361925 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "513e4049-c76b-4f14-9fe8-62323af1fbe8" (UID: "513e4049-c76b-4f14-9fe8-62323af1fbe8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:37:07.361988 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.361958 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "513e4049-c76b-4f14-9fe8-62323af1fbe8" (UID: "513e4049-c76b-4f14-9fe8-62323af1fbe8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:37:07.460316 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.460294 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:37:07.460316 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.460312 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-offline-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:37:07.460435 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.460320 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/513e4049-c76b-4f14-9fe8-62323af1fbe8-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:37:07.460435 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.460328 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2lpz\" (UniqueName: \"kubernetes.io/projected/513e4049-c76b-4f14-9fe8-62323af1fbe8-kube-api-access-m2lpz\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:37:07.460435 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.460336 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-ui-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:37:07.460435 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.460344 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/513e4049-c76b-4f14-9fe8-62323af1fbe8-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:37:07.621175 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.621156 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt"] Jul 30 08:37:07.624831 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:07.624814 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qbsrt"] Jul 30 08:37:08.290234 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:08.290207 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" path="/var/lib/kubelet/pods/513e4049-c76b-4f14-9fe8-62323af1fbe8/volumes" Jul 30 08:37:58.219442 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219404 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4"] Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219686 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="offline" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219696 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="offline" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219713 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="feast-apply" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219719 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="feast-apply" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219727 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="registry" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219733 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="registry" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219742 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="feast-init" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219748 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="feast-init" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219754 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="online" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219759 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="online" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219765 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="ui" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219770 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="ui" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219817 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="offline" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219825 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="registry" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219831 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="online" Jul 30 08:37:58.219834 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.219837 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="513e4049-c76b-4f14-9fe8-62323af1fbe8" containerName="ui" Jul 30 08:37:58.223947 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.223930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.226800 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.226780 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 30 08:37:58.227157 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.227140 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-8c2tg\"" Jul 30 08:37:58.227317 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.227298 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 30 08:37:58.227480 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.227459 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 30 08:37:58.227547 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.227492 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 30 08:37:58.227703 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.227688 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 30 08:37:58.227927 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.227910 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 30 08:37:58.234405 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.234362 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4"] Jul 30 08:37:58.295687 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.295664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.295843 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.295693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.295843 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.295710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.295843 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.295791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp9cr\" (UniqueName: \"kubernetes.io/projected/fe37400b-fc56-4270-a706-9792a6371d7f-kube-api-access-tp9cr\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.295843 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.295828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fe37400b-fc56-4270-a706-9792a6371d7f-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.296005 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.295853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/fe37400b-fc56-4270-a706-9792a6371d7f-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.396641 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.396613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.396641 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.396646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.396884 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.396663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.396884 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.396689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9cr\" (UniqueName: \"kubernetes.io/projected/fe37400b-fc56-4270-a706-9792a6371d7f-kube-api-access-tp9cr\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.396884 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.396711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fe37400b-fc56-4270-a706-9792a6371d7f-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.396884 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.396728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/fe37400b-fc56-4270-a706-9792a6371d7f-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.397140 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.397121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fe37400b-fc56-4270-a706-9792a6371d7f-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.397442 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.397419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/fe37400b-fc56-4270-a706-9792a6371d7f-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.399069 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.399042 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.399069 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.399063 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.399188 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.399096 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.404217 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.404196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp9cr\" (UniqueName: \"kubernetes.io/projected/fe37400b-fc56-4270-a706-9792a6371d7f-kube-api-access-tp9cr\") pod \"feast-simple-feast-remote-setup-db6955c6c-5kpl4\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.532610 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.532539 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:37:58.660444 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:58.660422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4"] Jul 30 08:37:58.662539 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:37:58.662502 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe37400b_fc56_4270_a706_9792a6371d7f.slice/crio-1134f1cf8c407c38beec439229b159367d587a71db01c923b673e73c370640bb WatchSource:0}: Error finding container 1134f1cf8c407c38beec439229b159367d587a71db01c923b673e73c370640bb: Status 404 returned error can't find the container with id 1134f1cf8c407c38beec439229b159367d587a71db01c923b673e73c370640bb Jul 30 08:37:59.476602 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:59.476567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerStarted","Data":"b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e"} Jul 30 08:37:59.477070 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:37:59.476608 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerStarted","Data":"1134f1cf8c407c38beec439229b159367d587a71db01c923b673e73c370640bb"} Jul 30 08:38:03.491219 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:03.491154 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe37400b-fc56-4270-a706-9792a6371d7f" containerID="b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e" exitCode=0 Jul 30 08:38:03.491219 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:03.491198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerDied","Data":"b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e"} Jul 30 08:38:04.496505 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:04.496470 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerStarted","Data":"563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048"} Jul 30 08:38:09.518064 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:09.518035 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe37400b-fc56-4270-a706-9792a6371d7f" containerID="563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048" exitCode=0 Jul 30 08:38:09.518408 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:09.518110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerDied","Data":"563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048"} Jul 30 08:38:10.525932 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:10.525896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerStarted","Data":"f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0"} Jul 30 08:38:10.526425 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:10.525949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerStarted","Data":"25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc"} Jul 30 08:38:10.526425 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:10.525965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerStarted","Data":"952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa"} Jul 30 08:38:10.533015 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:10.532988 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:38:10.533169 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:10.533030 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:38:10.533169 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:10.533047 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:38:10.534465 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:10.534430 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.33:8816: connect: connection refused" Jul 30 08:38:10.534579 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:10.534428 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Jul 30 08:38:10.534642 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:10.534429 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 30 08:38:10.550777 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:10.550739 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podStartSLOduration=12.550723468 podStartE2EDuration="12.550723468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:38:10.54874136 +0000 UTC m=+1584.808687709" watchObservedRunningTime="2026-07-30 08:38:10.550723468 +0000 UTC m=+1584.810669804" Jul 30 08:38:13.533458 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:13.533402 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 30 08:38:13.533918 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:13.533395 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Jul 30 08:38:13.533918 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:13.533404 2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.33:8816: connect: connection refused" Jul 30 08:38:16.533429 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:16.533401 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:38:16.533929 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:16.533716 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:38:16.534011 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:16.533978 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:38:16.534070 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:16.534019 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:38:16.534489 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:16.534459 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:38:16.534629 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:16.534514 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:38:16.540881 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:16.540864 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:38:16.552825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:16.552807 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:38:16.557080 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:38:16.557063 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:39:17.245015 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:17.244976 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4"] Jul 30 08:39:17.245523 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:17.245267 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="online" containerID="cri-o://952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa" gracePeriod=30 Jul 30 08:39:17.245523 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:17.245331 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="ui" containerID="cri-o://f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0" gracePeriod=30 Jul 30 08:39:17.245523 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:17.245492 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="offline" containerID="cri-o://25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc" gracePeriod=30 Jul 30 08:39:17.388969 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:17.388937 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv"] Jul 30 08:39:17.389245 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:17.389161 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="registry" containerID="cri-o://806dc42fd81578d8ebf9772242816bcdd19f53bce16a073aeefeefd199201a80" gracePeriod=30 Jul 30 08:39:17.389405 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:17.389209 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="ui" containerID="cri-o://daf7cfc93e7e8e4084344d933bc1e4a7c1bfb7ba2bdc31f3882f8d26c3744c49" gracePeriod=30 Jul 30 08:39:17.389487 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:17.389211 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="offline" containerID="cri-o://f97522d1efdd194930bbb1e67d0f1e6574f1f14382ec6caabea161cca889f38c" gracePeriod=30 Jul 30 08:39:17.389545 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:17.389220 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="online" containerID="cri-o://b73acd50bf02f2c7436aa317cabb014d3c343f2d48a312c362e298fe9ff757f2" gracePeriod=30 Jul 30 08:39:17.755994 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:39:17.755951 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe37400b_fc56_4270_a706_9792a6371d7f.slice/crio-conmon-f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0.scope\": RecentStats: unable to find data in memory cache]" Jul 30 08:39:17.775975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:17.775941 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe37400b-fc56-4270-a706-9792a6371d7f" containerID="f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0" exitCode=0 Jul 30 08:39:17.776168 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:17.776006 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerDied","Data":"f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0"} Jul 30 08:39:18.782414 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:18.782364 2576 generic.go:356] "Generic (PLEG): container finished" podID="7401f3b1-c49d-419e-99b9-ce916679026f" containerID="daf7cfc93e7e8e4084344d933bc1e4a7c1bfb7ba2bdc31f3882f8d26c3744c49" exitCode=0 Jul 30 08:39:18.782414 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:18.782408 2576 generic.go:356] "Generic (PLEG): container finished" podID="7401f3b1-c49d-419e-99b9-ce916679026f" containerID="b73acd50bf02f2c7436aa317cabb014d3c343f2d48a312c362e298fe9ff757f2" exitCode=0 Jul 30 08:39:18.782825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:18.782435 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerDied","Data":"daf7cfc93e7e8e4084344d933bc1e4a7c1bfb7ba2bdc31f3882f8d26c3744c49"} Jul 30 08:39:18.782825 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:18.782477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerDied","Data":"b73acd50bf02f2c7436aa317cabb014d3c343f2d48a312c362e298fe9ff757f2"} Jul 30 08:39:18.785230 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:18.785204 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe37400b-fc56-4270-a706-9792a6371d7f" containerID="952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa" exitCode=0 Jul 30 08:39:18.785346 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:18.785272 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerDied","Data":"952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa"} Jul 30 08:39:26.256662 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:26.256611 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Jul 30 08:39:26.257094 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:26.256672 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 30 08:39:26.534648 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:26.534558 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Jul 30 08:39:26.552975 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:26.552935 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 30 08:39:36.256585 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:36.256532 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Jul 30 08:39:36.257018 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:36.256595 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 30 08:39:36.534835 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:36.534749 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Jul 30 08:39:36.552882 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:36.552850 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 30 08:39:46.255838 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:46.255798 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Jul 30 08:39:46.255838 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:46.255822 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 30 08:39:46.256542 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:46.255903 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:39:46.256542 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:46.256063 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:39:46.534366 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:46.534263 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Jul 30 08:39:46.534585 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:46.534406 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:39:46.552927 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:46.552899 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 30 08:39:46.553096 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:46.553006 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:39:47.520193 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.520143 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:39:47.555124 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.555106 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/fe37400b-fc56-4270-a706-9792a6371d7f-registry-tls\") pod \"fe37400b-fc56-4270-a706-9792a6371d7f\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " Jul 30 08:39:47.555244 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.555139 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-ui-tls\") pod \"fe37400b-fc56-4270-a706-9792a6371d7f\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " Jul 30 08:39:47.555244 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.555173 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-offline-tls\") pod \"fe37400b-fc56-4270-a706-9792a6371d7f\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " Jul 30 08:39:47.555244 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.555190 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp9cr\" (UniqueName: \"kubernetes.io/projected/fe37400b-fc56-4270-a706-9792a6371d7f-kube-api-access-tp9cr\") pod \"fe37400b-fc56-4270-a706-9792a6371d7f\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " Jul 30 08:39:47.555411 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.555306 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fe37400b-fc56-4270-a706-9792a6371d7f-feast-data\") pod \"fe37400b-fc56-4270-a706-9792a6371d7f\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " Jul 30 08:39:47.555411 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.555356 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-online-tls\") pod \"fe37400b-fc56-4270-a706-9792a6371d7f\" (UID: \"fe37400b-fc56-4270-a706-9792a6371d7f\") " Jul 30 08:39:47.555596 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.555510 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe37400b-fc56-4270-a706-9792a6371d7f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fe37400b-fc56-4270-a706-9792a6371d7f" (UID: "fe37400b-fc56-4270-a706-9792a6371d7f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 08:39:47.555669 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.555651 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/fe37400b-fc56-4270-a706-9792a6371d7f-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:47.555853 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.555826 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe37400b-fc56-4270-a706-9792a6371d7f-feast-data" (OuterVolumeSpecName: "feast-data") pod "fe37400b-fc56-4270-a706-9792a6371d7f" (UID: "fe37400b-fc56-4270-a706-9792a6371d7f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:39:47.557696 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.557660 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe37400b-fc56-4270-a706-9792a6371d7f-kube-api-access-tp9cr" (OuterVolumeSpecName: "kube-api-access-tp9cr") pod "fe37400b-fc56-4270-a706-9792a6371d7f" (UID: "fe37400b-fc56-4270-a706-9792a6371d7f"). InnerVolumeSpecName "kube-api-access-tp9cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:39:47.557784 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.557728 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "fe37400b-fc56-4270-a706-9792a6371d7f" (UID: "fe37400b-fc56-4270-a706-9792a6371d7f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:39:47.557784 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.557738 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "fe37400b-fc56-4270-a706-9792a6371d7f" (UID: "fe37400b-fc56-4270-a706-9792a6371d7f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:39:47.557784 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.557753 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-online-tls" (OuterVolumeSpecName: "online-tls") pod "fe37400b-fc56-4270-a706-9792a6371d7f" (UID: "fe37400b-fc56-4270-a706-9792a6371d7f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:39:47.656207 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.656180 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-ui-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:47.656207 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.656205 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-offline-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:47.656207 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.656214 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp9cr\" (UniqueName: \"kubernetes.io/projected/fe37400b-fc56-4270-a706-9792a6371d7f-kube-api-access-tp9cr\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:47.656355 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.656224 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fe37400b-fc56-4270-a706-9792a6371d7f-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:47.656355 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.656250 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fe37400b-fc56-4270-a706-9792a6371d7f-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:47.892822 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.892797 2576 generic.go:356] "Generic (PLEG): container finished" podID="7401f3b1-c49d-419e-99b9-ce916679026f" containerID="f97522d1efdd194930bbb1e67d0f1e6574f1f14382ec6caabea161cca889f38c" exitCode=137 Jul 30 08:39:47.892822 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.892818 2576 generic.go:356] "Generic (PLEG): container finished" podID="7401f3b1-c49d-419e-99b9-ce916679026f" containerID="806dc42fd81578d8ebf9772242816bcdd19f53bce16a073aeefeefd199201a80" exitCode=137 Jul 30 08:39:47.892973 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.892864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerDied","Data":"f97522d1efdd194930bbb1e67d0f1e6574f1f14382ec6caabea161cca889f38c"} Jul 30 08:39:47.892973 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.892904 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerDied","Data":"806dc42fd81578d8ebf9772242816bcdd19f53bce16a073aeefeefd199201a80"} Jul 30 08:39:47.894963 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.894944 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe37400b-fc56-4270-a706-9792a6371d7f" containerID="25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc" exitCode=137 Jul 30 08:39:47.895049 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.895001 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" Jul 30 08:39:47.895049 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.895028 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerDied","Data":"25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc"} Jul 30 08:39:47.895116 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.895070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4" event={"ID":"fe37400b-fc56-4270-a706-9792a6371d7f","Type":"ContainerDied","Data":"1134f1cf8c407c38beec439229b159367d587a71db01c923b673e73c370640bb"} Jul 30 08:39:47.895116 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.895094 2576 scope.go:117] "RemoveContainer" containerID="f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0" Jul 30 08:39:47.903599 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.903583 2576 scope.go:117] "RemoveContainer" containerID="25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc" Jul 30 08:39:47.910626 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.910605 2576 scope.go:117] "RemoveContainer" containerID="952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa" Jul 30 08:39:47.916646 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.916627 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4"] Jul 30 08:39:47.917630 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.917613 2576 scope.go:117] "RemoveContainer" containerID="563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048" Jul 30 08:39:47.920195 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.920175 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5kpl4"] Jul 30 08:39:47.924995 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.924975 2576 scope.go:117] "RemoveContainer" containerID="b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e" Jul 30 08:39:47.931160 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.931141 2576 scope.go:117] "RemoveContainer" containerID="f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0" Jul 30 08:39:47.931434 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:39:47.931416 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0\": container with ID starting with f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0 not found: ID does not exist" containerID="f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0" Jul 30 08:39:47.931509 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.931440 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0"} err="failed to get container status \"f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0\": rpc error: code = NotFound desc = could not find container \"f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0\": container with ID starting with f76a33276e8d2c7f01558d064b32cfba877a22154edcfbabf6fb6b5c55b310e0 not found: ID does not exist" Jul 30 08:39:47.931509 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.931462 2576 scope.go:117] "RemoveContainer" containerID="25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc" Jul 30 08:39:47.931699 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:39:47.931682 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc\": container with ID starting with 25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc not found: ID does not exist" containerID="25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc" Jul 30 08:39:47.931742 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.931712 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc"} err="failed to get container status \"25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc\": rpc error: code = NotFound desc = could not find container \"25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc\": container with ID starting with 25141c854aa6763649d6c7c042881e96cd7dca3ce86bd3c556a11ab81f49c4cc not found: ID does not exist" Jul 30 08:39:47.931742 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.931727 2576 scope.go:117] "RemoveContainer" containerID="952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa" Jul 30 08:39:47.931933 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:39:47.931916 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa\": container with ID starting with 952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa not found: ID does not exist" containerID="952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa" Jul 30 08:39:47.931999 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.931939 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa"} err="failed to get container status \"952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa\": rpc error: code = NotFound desc = could not find container \"952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa\": container with ID starting with 952c09f6b1f88b6f64e6843ba21040062e26d5fd4bef455a810bb10b60ecffaa not found: ID does not exist" Jul 30 08:39:47.931999 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.931959 2576 scope.go:117] "RemoveContainer" containerID="563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048" Jul 30 08:39:47.932184 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:39:47.932168 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048\": container with ID starting with 563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048 not found: ID does not exist" containerID="563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048" Jul 30 08:39:47.932239 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.932190 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048"} err="failed to get container status \"563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048\": rpc error: code = NotFound desc = could not find container \"563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048\": container with ID starting with 563a66818626406144dfe22e92d6cf90615b472b65b0a398d3d54537b5803048 not found: ID does not exist" Jul 30 08:39:47.932239 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.932209 2576 scope.go:117] "RemoveContainer" containerID="b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e" Jul 30 08:39:47.932463 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:39:47.932440 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e\": container with ID starting with b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e not found: ID does not exist" containerID="b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e" Jul 30 08:39:47.932530 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:47.932466 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e"} err="failed to get container status \"b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e\": rpc error: code = NotFound desc = could not find container \"b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e\": container with ID starting with b7eb6674cd0a97140df973b3ee0113dc6d139c933a6a3b4eb605548165a0384e not found: ID does not exist" Jul 30 08:39:48.031509 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.031486 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:39:48.058581 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.058559 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-online-tls\") pod \"7401f3b1-c49d-419e-99b9-ce916679026f\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " Jul 30 08:39:48.058686 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.058593 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-offline-tls\") pod \"7401f3b1-c49d-419e-99b9-ce916679026f\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " Jul 30 08:39:48.058686 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.058609 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-ui-tls\") pod \"7401f3b1-c49d-419e-99b9-ce916679026f\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " Jul 30 08:39:48.058686 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.058664 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s526x\" (UniqueName: \"kubernetes.io/projected/7401f3b1-c49d-419e-99b9-ce916679026f-kube-api-access-s526x\") pod \"7401f3b1-c49d-419e-99b9-ce916679026f\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " Jul 30 08:39:48.058686 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.058685 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7401f3b1-c49d-419e-99b9-ce916679026f-feast-data\") pod \"7401f3b1-c49d-419e-99b9-ce916679026f\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " Jul 30 08:39:48.059252 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.059228 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7401f3b1-c49d-419e-99b9-ce916679026f-feast-data" (OuterVolumeSpecName: "feast-data") pod "7401f3b1-c49d-419e-99b9-ce916679026f" (UID: "7401f3b1-c49d-419e-99b9-ce916679026f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:39:48.061029 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.061001 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7401f3b1-c49d-419e-99b9-ce916679026f-kube-api-access-s526x" (OuterVolumeSpecName: "kube-api-access-s526x") pod "7401f3b1-c49d-419e-99b9-ce916679026f" (UID: "7401f3b1-c49d-419e-99b9-ce916679026f"). InnerVolumeSpecName "kube-api-access-s526x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:39:48.061113 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.061044 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7401f3b1-c49d-419e-99b9-ce916679026f" (UID: "7401f3b1-c49d-419e-99b9-ce916679026f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:39:48.061113 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.061069 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-online-tls" (OuterVolumeSpecName: "online-tls") pod "7401f3b1-c49d-419e-99b9-ce916679026f" (UID: "7401f3b1-c49d-419e-99b9-ce916679026f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:39:48.061113 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.061085 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "7401f3b1-c49d-419e-99b9-ce916679026f" (UID: "7401f3b1-c49d-419e-99b9-ce916679026f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:39:48.159437 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.159387 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-registry-tls\") pod \"7401f3b1-c49d-419e-99b9-ce916679026f\" (UID: \"7401f3b1-c49d-419e-99b9-ce916679026f\") " Jul 30 08:39:48.159517 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.159497 2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7401f3b1-c49d-419e-99b9-ce916679026f-feast-data\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:48.159517 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.159508 2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-online-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:48.159517 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.159515 2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-offline-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:48.159611 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.159522 2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-ui-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:48.159611 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.159531 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s526x\" (UniqueName: \"kubernetes.io/projected/7401f3b1-c49d-419e-99b9-ce916679026f-kube-api-access-s526x\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:48.160903 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.160885 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7401f3b1-c49d-419e-99b9-ce916679026f" (UID: "7401f3b1-c49d-419e-99b9-ce916679026f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 08:39:48.260250 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.260227 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7401f3b1-c49d-419e-99b9-ce916679026f-registry-tls\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:39:48.290534 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.290512 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" path="/var/lib/kubelet/pods/fe37400b-fc56-4270-a706-9792a6371d7f/volumes" Jul 30 08:39:48.901506 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.901471 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" event={"ID":"7401f3b1-c49d-419e-99b9-ce916679026f","Type":"ContainerDied","Data":"2e57c2e9bfcd59e1b8c7eb278f822de61c3beb3445f06bff3b8f8beb29d5f9fa"} Jul 30 08:39:48.901884 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.901516 2576 scope.go:117] "RemoveContainer" containerID="daf7cfc93e7e8e4084344d933bc1e4a7c1bfb7ba2bdc31f3882f8d26c3744c49" Jul 30 08:39:48.901884 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.901480 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv" Jul 30 08:39:48.909226 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.909207 2576 scope.go:117] "RemoveContainer" containerID="f97522d1efdd194930bbb1e67d0f1e6574f1f14382ec6caabea161cca889f38c" Jul 30 08:39:48.916090 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.916073 2576 scope.go:117] "RemoveContainer" containerID="b73acd50bf02f2c7436aa317cabb014d3c343f2d48a312c362e298fe9ff757f2" Jul 30 08:39:48.920499 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.920479 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv"] Jul 30 08:39:48.923778 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.923760 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppkdv"] Jul 30 08:39:48.923839 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.923801 2576 scope.go:117] "RemoveContainer" containerID="806dc42fd81578d8ebf9772242816bcdd19f53bce16a073aeefeefd199201a80" Jul 30 08:39:48.930329 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.930314 2576 scope.go:117] "RemoveContainer" containerID="bac68a7df43563f1a2b8aa77d2fe23c36c2421761dd8d2d5945ea185f6949548" Jul 30 08:39:48.938159 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:48.937936 2576 scope.go:117] "RemoveContainer" containerID="5f0073d940bde409bf5c8799c27e3e96e7ffb4615f8028bcd421530508f2979c" Jul 30 08:39:50.290714 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:39:50.290683 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" path="/var/lib/kubelet/pods/7401f3b1-c49d-419e-99b9-ce916679026f/volumes" Jul 30 08:40:06.560267 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560234 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4zpkg/must-gather-fcsvp"] Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560590 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="feast-apply" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560605 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="feast-apply" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560618 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="feast-apply" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560627 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="feast-apply" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560641 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="feast-init" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560648 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="feast-init" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560656 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="offline" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560661 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="offline" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560667 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="ui" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560672 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="ui" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560678 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="online" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560683 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="online" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560691 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="ui" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560699 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="ui" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560709 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="registry" Jul 30 08:40:06.560711 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560717 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="registry" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560728 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="online" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560737 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="online" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560748 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="feast-init" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560756 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="feast-init" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560762 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="offline" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560768 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="offline" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560814 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="offline" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560826 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="online" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560832 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="online" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560837 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="ui" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560845 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="ui" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560854 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe37400b-fc56-4270-a706-9792a6371d7f" containerName="offline" Jul 30 08:40:06.561330 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.560861 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7401f3b1-c49d-419e-99b9-ce916679026f" containerName="registry" Jul 30 08:40:06.566046 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.566026 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:06.568197 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.568179 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4zpkg\"/\"openshift-service-ca.crt\"" Jul 30 08:40:06.568197 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.568193 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4zpkg\"/\"default-dockercfg-ndm48\"" Jul 30 08:40:06.569443 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.569368 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4zpkg\"/\"kube-root-ca.crt\"" Jul 30 08:40:06.570611 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.570590 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4zpkg/must-gather-fcsvp"] Jul 30 08:40:06.677415 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.677396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-must-gather-output\") pod \"must-gather-fcsvp\" (UID: \"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58\") " pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:06.677520 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.677435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxg8\" (UniqueName: \"kubernetes.io/projected/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-kube-api-access-scxg8\") pod \"must-gather-fcsvp\" (UID: \"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58\") " pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:06.778161 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.778139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-must-gather-output\") pod \"must-gather-fcsvp\" (UID: \"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58\") " pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:06.778233 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.778180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxg8\" (UniqueName: \"kubernetes.io/projected/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-kube-api-access-scxg8\") pod \"must-gather-fcsvp\" (UID: \"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58\") " pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:06.778442 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.778427 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-must-gather-output\") pod \"must-gather-fcsvp\" (UID: \"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58\") " pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:06.786360 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.786339 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxg8\" (UniqueName: \"kubernetes.io/projected/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-kube-api-access-scxg8\") pod \"must-gather-fcsvp\" (UID: \"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58\") " pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:06.883468 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:06.883422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:07.003936 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:07.003909 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4zpkg/must-gather-fcsvp"] Jul 30 08:40:07.006521 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:40:07.006501 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc19146d3_0a2c_4ca2_91ac_9b126fdd3a58.slice/crio-1c778db6b1d03f534760db3ad45fe605c11c386c308c38909a608c95fab5817c WatchSource:0}: Error finding container 1c778db6b1d03f534760db3ad45fe605c11c386c308c38909a608c95fab5817c: Status 404 returned error can't find the container with id 1c778db6b1d03f534760db3ad45fe605c11c386c308c38909a608c95fab5817c Jul 30 08:40:07.008648 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:07.008635 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 08:40:07.978121 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:07.977881 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" event={"ID":"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58","Type":"ContainerStarted","Data":"1c778db6b1d03f534760db3ad45fe605c11c386c308c38909a608c95fab5817c"} Jul 30 08:40:14.004495 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:14.004453 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" event={"ID":"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58","Type":"ContainerStarted","Data":"b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d"} Jul 30 08:40:14.004495 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:14.004498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" event={"ID":"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58","Type":"ContainerStarted","Data":"b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0"} Jul 30 08:40:14.019907 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:14.019863 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" podStartSLOduration=2.119172928 podStartE2EDuration="8.019847986s" totalImagesPullingTime="5.900675058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:40:14.018624154 +0000 UTC m=+1708.278570491" watchObservedRunningTime="2026-07-30 08:40:14.019847986 +0000 UTC m=+1708.279794335" Jul 30 08:40:23.040399 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.040354 2576 generic.go:356] "Generic (PLEG): container finished" podID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" containerID="b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0" exitCode=0 Jul 30 08:40:23.040766 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.040420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" event={"ID":"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58","Type":"ContainerDied","Data":"b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0"} Jul 30 08:40:23.040766 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.040657 2576 scope.go:117] "RemoveContainer" containerID="b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0" Jul 30 08:40:23.209092 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.209069 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4zpkg_must-gather-fcsvp_c19146d3-0a2c-4ca2-91ac-9b126fdd3a58/gather/0.log" Jul 30 08:40:23.660076 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.660015 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bh2gv/must-gather-cr6ln"] Jul 30 08:40:23.662565 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.662551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bh2gv/must-gather-cr6ln" Jul 30 08:40:23.664627 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.664608 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bh2gv\"/\"default-dockercfg-zsxcb\"" Jul 30 08:40:23.664789 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.664770 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bh2gv\"/\"openshift-service-ca.crt\"" Jul 30 08:40:23.664882 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.664865 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bh2gv\"/\"kube-root-ca.crt\"" Jul 30 08:40:23.672272 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.672253 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bh2gv/must-gather-cr6ln"] Jul 30 08:40:23.709790 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.709766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa988977-449e-4504-b1c1-f5d46973eaba-must-gather-output\") pod \"must-gather-cr6ln\" (UID: \"fa988977-449e-4504-b1c1-f5d46973eaba\") " pod="openshift-must-gather-bh2gv/must-gather-cr6ln" Jul 30 08:40:23.709875 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.709809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dg7l\" (UniqueName: \"kubernetes.io/projected/fa988977-449e-4504-b1c1-f5d46973eaba-kube-api-access-4dg7l\") pod \"must-gather-cr6ln\" (UID: \"fa988977-449e-4504-b1c1-f5d46973eaba\") " pod="openshift-must-gather-bh2gv/must-gather-cr6ln" Jul 30 08:40:23.810900 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.810876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa988977-449e-4504-b1c1-f5d46973eaba-must-gather-output\") pod \"must-gather-cr6ln\" (UID: \"fa988977-449e-4504-b1c1-f5d46973eaba\") " pod="openshift-must-gather-bh2gv/must-gather-cr6ln" Jul 30 08:40:23.810982 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.810923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dg7l\" (UniqueName: \"kubernetes.io/projected/fa988977-449e-4504-b1c1-f5d46973eaba-kube-api-access-4dg7l\") pod \"must-gather-cr6ln\" (UID: \"fa988977-449e-4504-b1c1-f5d46973eaba\") " pod="openshift-must-gather-bh2gv/must-gather-cr6ln" Jul 30 08:40:23.811170 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.811155 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa988977-449e-4504-b1c1-f5d46973eaba-must-gather-output\") pod \"must-gather-cr6ln\" (UID: \"fa988977-449e-4504-b1c1-f5d46973eaba\") " pod="openshift-must-gather-bh2gv/must-gather-cr6ln" Jul 30 08:40:23.820435 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.820420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dg7l\" (UniqueName: \"kubernetes.io/projected/fa988977-449e-4504-b1c1-f5d46973eaba-kube-api-access-4dg7l\") pod \"must-gather-cr6ln\" (UID: \"fa988977-449e-4504-b1c1-f5d46973eaba\") " pod="openshift-must-gather-bh2gv/must-gather-cr6ln" Jul 30 08:40:23.970788 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:23.970742 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bh2gv/must-gather-cr6ln" Jul 30 08:40:24.092848 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:24.092823 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bh2gv/must-gather-cr6ln"] Jul 30 08:40:24.095236 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:40:24.095195 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa988977_449e_4504_b1c1_f5d46973eaba.slice/crio-69346940b66c886272638c5846e5140bfc77266100369f6c10636b205bbcc02a WatchSource:0}: Error finding container 69346940b66c886272638c5846e5140bfc77266100369f6c10636b205bbcc02a: Status 404 returned error can't find the container with id 69346940b66c886272638c5846e5140bfc77266100369f6c10636b205bbcc02a Jul 30 08:40:25.056058 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:25.056020 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bh2gv/must-gather-cr6ln" event={"ID":"fa988977-449e-4504-b1c1-f5d46973eaba","Type":"ContainerStarted","Data":"69346940b66c886272638c5846e5140bfc77266100369f6c10636b205bbcc02a"} Jul 30 08:40:26.061292 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:26.061256 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bh2gv/must-gather-cr6ln" event={"ID":"fa988977-449e-4504-b1c1-f5d46973eaba","Type":"ContainerStarted","Data":"62b7f129aae96d4f26130d5c600f69664da6e90b4bb1c397701ea96ebef6d75e"} Jul 30 08:40:26.061693 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:26.061305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bh2gv/must-gather-cr6ln" event={"ID":"fa988977-449e-4504-b1c1-f5d46973eaba","Type":"ContainerStarted","Data":"3c4c89e17677cca38c3d01f95b2322b3836bfbf7b8f28db0119c8e298e257ec9"} Jul 30 08:40:26.076654 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:26.076611 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bh2gv/must-gather-cr6ln" podStartSLOduration=2.00347304 podStartE2EDuration="3.076597262s" totalImagesPullingTime="1.073124222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 08:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:40:26.075300683 +0000 UTC m=+1720.335247012" watchObservedRunningTime="2026-07-30 08:40:26.076597262 +0000 UTC m=+1720.336543597" Jul 30 08:40:26.489057 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:26.488989 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wtt8f_da7e45bf-dfea-414e-8f0b-98242e6d10f5/global-pull-secret-syncer/0.log" Jul 30 08:40:26.560470 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:26.560435 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gz5j5_d1a8039a-aa83-4ab0-90a3-41d6ad89f288/konnectivity-agent/0.log" Jul 30 08:40:26.630130 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:26.630102 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-119.ec2.internal_f87618faf1b1c298cf86a88886360a92/haproxy/0.log" Jul 30 08:40:28.517453 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:28.517415 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-4zpkg/must-gather-fcsvp"] Jul 30 08:40:28.518039 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:28.517713 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" containerName="copy" containerID="cri-o://b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d" gracePeriod=2 Jul 30 08:40:28.520105 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:40:28.520067 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fcsvp\" is forbidden: User \"system:node:ip-10-0-136-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4zpkg\": no relationship found between node 'ip-10-0-136-119.ec2.internal' and this object" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:28.520617 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:28.520590 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-4zpkg/must-gather-fcsvp"] Jul 30 08:40:28.877820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:28.877754 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4zpkg_must-gather-fcsvp_c19146d3-0a2c-4ca2-91ac-9b126fdd3a58/copy/0.log" Jul 30 08:40:28.881550 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:28.881262 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:28.883342 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:40:28.883303 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fcsvp\" is forbidden: User \"system:node:ip-10-0-136-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4zpkg\": no relationship found between node 'ip-10-0-136-119.ec2.internal' and this object" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:28.971968 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:28.971075 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scxg8\" (UniqueName: \"kubernetes.io/projected/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-kube-api-access-scxg8\") pod \"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58\" (UID: \"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58\") " Jul 30 08:40:28.971968 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:28.971159 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-must-gather-output\") pod \"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58\" (UID: \"c19146d3-0a2c-4ca2-91ac-9b126fdd3a58\") " Jul 30 08:40:28.971968 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:28.971667 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" (UID: "c19146d3-0a2c-4ca2-91ac-9b126fdd3a58"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 08:40:28.981823 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:28.981777 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-kube-api-access-scxg8" (OuterVolumeSpecName: "kube-api-access-scxg8") pod "c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" (UID: "c19146d3-0a2c-4ca2-91ac-9b126fdd3a58"). InnerVolumeSpecName "kube-api-access-scxg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 08:40:29.072593 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:29.072531 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scxg8\" (UniqueName: \"kubernetes.io/projected/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-kube-api-access-scxg8\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:40:29.072593 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:29.072566 2576 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58-must-gather-output\") on node \"ip-10-0-136-119.ec2.internal\" DevicePath \"\"" Jul 30 08:40:29.079935 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:29.079700 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4zpkg_must-gather-fcsvp_c19146d3-0a2c-4ca2-91ac-9b126fdd3a58/copy/0.log" Jul 30 08:40:29.083403 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:29.080265 2576 generic.go:356] "Generic (PLEG): container finished" podID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" containerID="b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d" exitCode=143 Jul 30 08:40:29.083403 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:29.080334 2576 scope.go:117] "RemoveContainer" containerID="b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d" Jul 30 08:40:29.083403 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:29.080503 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:29.091658 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:40:29.091614 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fcsvp\" is forbidden: User \"system:node:ip-10-0-136-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4zpkg\": no relationship found between node 'ip-10-0-136-119.ec2.internal' and this object" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:29.098284 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:29.098230 2576 scope.go:117] "RemoveContainer" containerID="b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0" Jul 30 08:40:29.100199 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:40:29.100154 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fcsvp\" is forbidden: User \"system:node:ip-10-0-136-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4zpkg\": no relationship found between node 'ip-10-0-136-119.ec2.internal' and this object" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" pod="openshift-must-gather-4zpkg/must-gather-fcsvp" Jul 30 08:40:29.115610 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:29.115390 2576 scope.go:117] "RemoveContainer" containerID="b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d" Jul 30 08:40:29.115964 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:40:29.115930 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d\": container with ID starting with b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d not found: ID does not exist" containerID="b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d" Jul 30 08:40:29.116146 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:29.116090 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d"} err="failed to get container status \"b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d\": rpc error: code = NotFound desc = could not find container \"b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d\": container with ID starting with b07a0780951c8c4ebafcb70f90abaa0cd95c2e10029a5e2ffb24f0a84f72ef4d not found: ID does not exist" Jul 30 08:40:29.116146 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:29.116120 2576 scope.go:117] "RemoveContainer" containerID="b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0" Jul 30 08:40:29.116712 ip-10-0-136-119 kubenswrapper[2576]: E0730 08:40:29.116659 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0\": container with ID starting with b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0 not found: ID does not exist" containerID="b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0" Jul 30 08:40:29.116820 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:29.116716 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0"} err="failed to get container status \"b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0\": rpc error: code = NotFound desc = could not find container \"b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0\": container with ID starting with b822392e2a9e2373f0d97915acae1008c60bc3a39e23072100af02f557e052b0 not found: ID does not exist" Jul 30 08:40:30.293146 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:30.293104 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" path="/var/lib/kubelet/pods/c19146d3-0a2c-4ca2-91ac-9b126fdd3a58/volumes" Jul 30 08:40:30.520902 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:30.520830 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-stdlx_9a98b277-2607-4cac-bfe3-5023f1b78468/kube-state-metrics/0.log" Jul 30 08:40:30.542171 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:30.542137 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-stdlx_9a98b277-2607-4cac-bfe3-5023f1b78468/kube-rbac-proxy-main/0.log" Jul 30 08:40:30.565438 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:30.565398 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-stdlx_9a98b277-2607-4cac-bfe3-5023f1b78468/kube-rbac-proxy-self/0.log" Jul 30 08:40:30.832010 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:30.831982 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lp6qh_7e8fc961-8af7-4963-8646-e2820bf06413/node-exporter/0.log" Jul 30 08:40:30.858133 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:30.858106 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lp6qh_7e8fc961-8af7-4963-8646-e2820bf06413/kube-rbac-proxy/0.log" Jul 30 08:40:30.880610 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:30.880586 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lp6qh_7e8fc961-8af7-4963-8646-e2820bf06413/init-textfile/0.log" Jul 30 08:40:31.254219 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:31.254129 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-h455g_11949d1b-659e-43d2-9e04-c60edaf4bc45/prometheus-operator/0.log" Jul 30 08:40:31.277043 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:31.277017 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-h455g_11949d1b-659e-43d2-9e04-c60edaf4bc45/kube-rbac-proxy/0.log" Jul 30 08:40:32.689632 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:32.689602 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-vjbrh_175def52-ca03-46f0-8b82-ee15bb501ee6/networking-console-plugin/0.log" Jul 30 08:40:33.068233 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.068211 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/1.log" Jul 30 08:40:33.073959 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.073939 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c9mnf_9e1bf430-3c04-4542-87b6-f0f29509e277/console-operator/2.log" Jul 30 08:40:33.461014 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.460896 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt"] Jul 30 08:40:33.461317 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.461294 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" containerName="gather" Jul 30 08:40:33.461317 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.461315 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" containerName="gather" Jul 30 08:40:33.461317 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.461322 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" containerName="copy" Jul 30 08:40:33.461519 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.461328 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" containerName="copy" Jul 30 08:40:33.461519 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.461394 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" containerName="copy" Jul 30 08:40:33.461519 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.461406 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19146d3-0a2c-4ca2-91ac-9b126fdd3a58" containerName="gather" Jul 30 08:40:33.465292 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.465270 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.472339 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.471997 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt"] Jul 30 08:40:33.611782 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.611739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-lib-modules\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.611990 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.611796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-sys\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.611990 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.611844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crvm\" (UniqueName: \"kubernetes.io/projected/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-kube-api-access-7crvm\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.611990 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.611884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-proc\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.611990 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.611938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-podres\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.712727 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.712646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-proc\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.712727 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.712719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-podres\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.713172 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.712770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-proc\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.713172 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.712787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-lib-modules\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.713172 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.712811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-sys\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.713172 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.712856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crvm\" (UniqueName: \"kubernetes.io/projected/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-kube-api-access-7crvm\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.713172 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.712908 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-podres\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.713172 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.712924 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-sys\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.713172 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.712928 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-lib-modules\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.721918 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.721891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crvm\" (UniqueName: \"kubernetes.io/projected/8b02c167-bd8e-4650-becb-3dfb7a62f9c1-kube-api-access-7crvm\") pod \"perf-node-gather-daemonset-9k2wt\" (UID: \"8b02c167-bd8e-4650-becb-3dfb7a62f9c1\") " pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.778818 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.778793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:33.938405 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:33.937629 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt"] Jul 30 08:40:33.941819 ip-10-0-136-119 kubenswrapper[2576]: W0730 08:40:33.941788 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8b02c167_bd8e_4650_becb_3dfb7a62f9c1.slice/crio-9f04d8436a9be6482e64be7ba184fd9a87e091ff988172afda7db1b46e7df62c WatchSource:0}: Error finding container 9f04d8436a9be6482e64be7ba184fd9a87e091ff988172afda7db1b46e7df62c: Status 404 returned error can't find the container with id 9f04d8436a9be6482e64be7ba184fd9a87e091ff988172afda7db1b46e7df62c Jul 30 08:40:34.116121 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:34.116092 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" event={"ID":"8b02c167-bd8e-4650-becb-3dfb7a62f9c1","Type":"ContainerStarted","Data":"cffd22dac8ce5fa8e87235c024db641e8804119d37ff32a6b0d435b044b2a807"} Jul 30 08:40:34.116236 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:34.116137 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" event={"ID":"8b02c167-bd8e-4650-becb-3dfb7a62f9c1","Type":"ContainerStarted","Data":"9f04d8436a9be6482e64be7ba184fd9a87e091ff988172afda7db1b46e7df62c"} Jul 30 08:40:34.116236 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:34.116191 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:34.133198 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:34.133159 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" podStartSLOduration=1.133146693 podStartE2EDuration="1.133146693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 08:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 08:40:34.13222936 +0000 UTC m=+1728.392175707" watchObservedRunningTime="2026-07-30 08:40:34.133146693 +0000 UTC m=+1728.393093038" Jul 30 08:40:34.738056 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:34.738023 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qfcjq_4fa47875-28d6-4eb9-86a1-1f9747488d9c/dns/0.log" Jul 30 08:40:34.761811 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:34.761790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qfcjq_4fa47875-28d6-4eb9-86a1-1f9747488d9c/kube-rbac-proxy/0.log" Jul 30 08:40:34.899200 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:34.899178 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lp4ct_bd9d1bb7-c615-40e0-b53a-d2ca26386d62/dns-node-resolver/0.log" Jul 30 08:40:35.287802 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:35.287773 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55b5bd6c4d-6xpbk_9077097e-03b5-409f-a035-9f6a2614bb55/registry/0.log" Jul 30 08:40:35.351684 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:35.351659 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s6r42_ab4c1b36-d20a-4ac3-96f3-fcd94d257061/node-ca/0.log" Jul 30 08:40:35.988410 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:35.988367 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68585dd8ff-bhdlw_b835a639-ac08-499a-9b99-79744d0e8e1b/router/0.log" Jul 30 08:40:36.347517 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:36.347487 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kdwtn_a2e7232f-ba93-4ce5-9815-9266879f62a0/serve-healthcheck-canary/0.log" Jul 30 08:40:36.656510 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:36.656442 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-75lsm_0ed27260-2f69-4fba-8783-e7155daa29e5/insights-operator/1.log" Jul 30 08:40:36.656649 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:36.656586 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-75lsm_0ed27260-2f69-4fba-8783-e7155daa29e5/insights-operator/0.log" Jul 30 08:40:36.675859 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:36.675837 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4twmv_7069b820-c097-4267-94ed-7c7d5ca4f8d3/kube-rbac-proxy/0.log" Jul 30 08:40:36.697064 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:36.697044 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4twmv_7069b820-c097-4267-94ed-7c7d5ca4f8d3/exporter/0.log" Jul 30 08:40:36.718741 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:36.718721 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4twmv_7069b820-c097-4267-94ed-7c7d5ca4f8d3/extractor/0.log" Jul 30 08:40:40.132830 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:40.132801 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bh2gv/perf-node-gather-daemonset-9k2wt" Jul 30 08:40:40.620081 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:40.620051 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-8qtn7_38c45b61-22fb-4184-8653-f624b4fe9761/migrator/0.log" Jul 30 08:40:40.653618 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:40.653596 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-8qtn7_38c45b61-22fb-4184-8653-f624b4fe9761/graceful-termination/0.log" Jul 30 08:40:41.780890 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:41.780866 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4zmjm_6d56f200-9b41-41c5-b48f-318c7d379588/kube-multus/0.log" Jul 30 08:40:42.398958 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:42.398932 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zpgxk_67e8b7f3-fb6e-4551-ae93-189127b4eea7/kube-multus-additional-cni-plugins/0.log" Jul 30 08:40:42.441176 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:42.441146 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zpgxk_67e8b7f3-fb6e-4551-ae93-189127b4eea7/egress-router-binary-copy/0.log" Jul 30 08:40:42.469355 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:42.469330 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zpgxk_67e8b7f3-fb6e-4551-ae93-189127b4eea7/cni-plugins/0.log" Jul 30 08:40:42.490181 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:42.490156 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zpgxk_67e8b7f3-fb6e-4551-ae93-189127b4eea7/bond-cni-plugin/0.log" Jul 30 08:40:42.511772 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:42.511745 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zpgxk_67e8b7f3-fb6e-4551-ae93-189127b4eea7/routeoverride-cni/0.log" Jul 30 08:40:42.541363 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:42.541334 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zpgxk_67e8b7f3-fb6e-4551-ae93-189127b4eea7/whereabouts-cni-bincopy/0.log" Jul 30 08:40:42.564902 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:42.564869 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zpgxk_67e8b7f3-fb6e-4551-ae93-189127b4eea7/whereabouts-cni/0.log" Jul 30 08:40:42.712859 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:42.712780 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rkhqs_f661b99d-6531-403f-8622-dec42ada1174/network-metrics-daemon/0.log" Jul 30 08:40:42.733183 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:42.733152 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rkhqs_f661b99d-6531-403f-8622-dec42ada1174/kube-rbac-proxy/0.log" Jul 30 08:40:43.794855 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:43.794828 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-controller/0.log" Jul 30 08:40:43.814237 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:43.814215 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/1.log" Jul 30 08:40:43.814418 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:43.814266 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovn-acl-logging/0.log" Jul 30 08:40:43.833850 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:43.833827 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/kube-rbac-proxy-node/0.log" Jul 30 08:40:43.869624 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:43.869601 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/kube-rbac-proxy-ovn-metrics/0.log" Jul 30 08:40:43.895389 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:43.895343 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/northd/0.log" Jul 30 08:40:43.946022 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:43.945996 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/nbdb/0.log" Jul 30 08:40:43.979411 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:43.979391 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/sbdb/0.log" Jul 30 08:40:44.026762 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:44.026706 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzfgq_f5ed960c-21db-4b02-a09a-122c1c90362e/ovnkube-controller/0.log" Jul 30 08:40:44.988193 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:44.988165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-xp7kt_b8c84511-af0e-41e8-bbaf-0cb318a69dce/network-check-target-container/0.log" Jul 30 08:40:45.820631 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:45.820599 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-lhbgs_6b832196-8be8-4c5f-aaf8-d81d73659e8b/iptables-alerter/0.log" Jul 30 08:40:46.521686 ip-10-0-136-119 kubenswrapper[2576]: I0730 08:40:46.521656 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-v6sg4_d25a97b3-406f-4e5b-a4b1-1bc1770ac7b6/tuned/0.log"