Sep 07 11:00:26.022744 ip-10-0-138-34 systemd[1]: Starting Kubernetes Kubelet... Sep 07 11:00:26.481436 ip-10-0-138-34 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 11:00:26.481436 ip-10-0-138-34 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 07 11:00:26.481436 ip-10-0-138-34 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 11:00:26.481436 ip-10-0-138-34 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 07 11:00:26.481436 ip-10-0-138-34 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 11:00:26.483593 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.483507 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 07 11:00:26.487745 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487728 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 11:00:26.487745 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487744 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487748 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487752 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487755 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487758 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487761 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487764 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487766 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487769 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487772 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487775 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487777 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487780 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487783 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487785 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487788 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487790 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487793 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487795 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487798 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 11:00:26.487813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487801 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487804 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487807 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487809 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487812 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487815 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487817 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487820 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487823 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487825 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487828 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487832 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487834 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487837 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487840 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487842 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487845 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487848 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487851 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487854 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 11:00:26.488274 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487858 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487860 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487863 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487865 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487868 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487870 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487873 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487875 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487879 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487883 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487886 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487889 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487891 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487894 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487898 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487901 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487904 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487906 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487909 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487912 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 11:00:26.488769 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487914 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487917 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487920 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487923 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487926 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487929 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487931 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487934 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487936 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487939 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487941 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487944 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487946 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487949 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487951 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487954 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487956 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487959 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487961 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 11:00:26.489244 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487964 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487966 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487972 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487975 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487979 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487982 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487986 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487989 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487992 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487994 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.487998 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488000 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488003 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488006 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488008 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488011 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488013 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488016 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488020 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 11:00:26.489739 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488715 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488721 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488724 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488726 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488730 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488732 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488735 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488738 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488741 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488743 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488746 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488749 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488751 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488754 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488758 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488762 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488764 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488767 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488770 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 11:00:26.490200 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488779 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488782 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488784 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488787 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488790 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488793 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488795 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488798 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488800 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488803 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488806 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488808 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488811 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488814 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488816 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488819 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488822 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488824 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488827 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 11:00:26.490812 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488830 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488833 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488835 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488838 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488840 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488843 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488846 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488848 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488851 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488883 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488887 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488891 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488894 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488898 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488901 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488903 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488906 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488908 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488911 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488914 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 11:00:26.491554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488916 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488919 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488921 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488924 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488926 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488928 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488931 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488935 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488940 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488943 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488946 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488949 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488951 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488953 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488956 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488959 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488961 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488963 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488966 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488969 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 11:00:26.492182 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488971 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488974 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488977 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488980 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488982 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488986 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488989 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488992 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488995 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.488997 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489000 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489002 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489005 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489007 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489010 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489012 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489015 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489017 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489020 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489022 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 11:00:26.492686 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489027 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489405 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489409 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489413 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489416 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489419 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489422 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489425 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489428 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489431 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489434 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489436 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489439 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489442 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489459 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489461 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489465 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489470 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489472 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 11:00:26.493152 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489475 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489478 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489481 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489483 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489486 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489488 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489491 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489493 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489498 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489501 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489504 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489506 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489509 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489512 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489514 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489517 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489519 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489522 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489525 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 11:00:26.493607 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489529 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489532 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489535 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489538 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489540 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489543 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489546 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489549 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489551 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489554 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489556 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489560 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489563 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489565 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489568 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489570 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489573 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489575 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489577 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489580 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 11:00:26.494059 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489582 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489586 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489589 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489592 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489594 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489597 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489599 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489602 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489605 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489607 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489609 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489614 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489616 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489619 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489621 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489624 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489626 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489629 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489631 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489634 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 11:00:26.494554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489636 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489638 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489641 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489645 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489647 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489649 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489652 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489654 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489657 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489659 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489662 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489664 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489666 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489669 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489671 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489674 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489676 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489678 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489681 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489683 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 11:00:26.495167 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489686 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.489688 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489765 2571 flags.go:64] FLAG: --address="0.0.0.0" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489800 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489806 2571 flags.go:64] FLAG: --anonymous-auth="true" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489810 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489814 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489818 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489822 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489830 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489833 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489837 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489840 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489843 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489846 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489864 2571 flags.go:64] FLAG: --cgroup-root="" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489868 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489871 2571 flags.go:64] FLAG: --client-ca-file="" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489874 2571 flags.go:64] FLAG: --cloud-provider="external" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489877 2571 flags.go:64] FLAG: --cluster-dns="[]" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489881 2571 flags.go:64] FLAG: --cluster-domain="" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489885 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489887 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489891 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 07 11:00:26.495662 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489894 2571 flags.go:64] FLAG: --container-log-max-files="5" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489898 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489901 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489904 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489907 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489910 2571 flags.go:64] FLAG: --contention-profiling="false" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489913 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489916 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489920 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489926 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489930 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489933 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489936 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489939 2571 flags.go:64] FLAG: --enable-load-reader="false" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489942 2571 flags.go:64] FLAG: --enable-server="true" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489946 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489950 2571 flags.go:64] FLAG: --event-burst="100" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489953 2571 flags.go:64] FLAG: --event-qps="50" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489956 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489959 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489962 2571 flags.go:64] FLAG: --eviction-hard="" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489965 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489969 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489972 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489975 2571 flags.go:64] FLAG: --eviction-soft="" Sep 07 11:00:26.496223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489978 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489981 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489984 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489987 2571 flags.go:64] FLAG: --experimental-mounter-path="" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489990 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489992 2571 flags.go:64] FLAG: --fail-swap-on="true" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489995 2571 flags.go:64] FLAG: --feature-gates="" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.489999 2571 flags.go:64] FLAG: --file-check-frequency="20s" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490002 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490005 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490008 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490011 2571 flags.go:64] FLAG: --healthz-port="10248" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490014 2571 flags.go:64] FLAG: --help="false" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490018 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490021 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490024 2571 flags.go:64] FLAG: --http-check-frequency="20s" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490027 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490031 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490034 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490037 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490040 2571 flags.go:64] FLAG: --image-service-endpoint="" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490043 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490046 2571 flags.go:64] FLAG: --kube-api-burst="100" Sep 07 11:00:26.496853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490049 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490052 2571 flags.go:64] FLAG: --kube-api-qps="50" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490054 2571 flags.go:64] FLAG: --kube-reserved="" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490057 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490060 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490063 2571 flags.go:64] FLAG: --kubelet-cgroups="" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490066 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490069 2571 flags.go:64] FLAG: --lock-file="" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490071 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490074 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490077 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490083 2571 flags.go:64] FLAG: --log-json-split-stream="false" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490085 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490088 2571 flags.go:64] FLAG: --log-text-split-stream="false" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490091 2571 flags.go:64] FLAG: --logging-format="text" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490094 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490098 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490101 2571 flags.go:64] FLAG: --manifest-url="" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490104 2571 flags.go:64] FLAG: --manifest-url-header="" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490108 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490111 2571 flags.go:64] FLAG: --max-open-files="1000000" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490115 2571 flags.go:64] FLAG: --max-pods="110" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490118 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490121 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490124 2571 flags.go:64] FLAG: --memory-manager-policy="None" Sep 07 11:00:26.497425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490127 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490130 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490133 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490136 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490143 2571 flags.go:64] FLAG: --node-status-max-images="50" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490146 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490149 2571 flags.go:64] FLAG: --oom-score-adj="-999" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490152 2571 flags.go:64] FLAG: --pod-cidr="" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490154 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490160 2571 flags.go:64] FLAG: --pod-manifest-path="" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490163 2571 flags.go:64] FLAG: --pod-max-pids="-1" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490166 2571 flags.go:64] FLAG: --pods-per-core="0" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490169 2571 flags.go:64] FLAG: --port="10250" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490172 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490175 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f21e15cd08a2434f" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490178 2571 flags.go:64] FLAG: --qos-reserved="" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490181 2571 flags.go:64] FLAG: --read-only-port="10255" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490184 2571 flags.go:64] FLAG: --register-node="true" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490187 2571 flags.go:64] FLAG: --register-with-taints="" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490190 2571 flags.go:64] FLAG: --registry-burst="10" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490193 2571 flags.go:64] FLAG: --registry-qps="5" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490196 2571 flags.go:64] FLAG: --reserved-cpus="" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490199 2571 flags.go:64] FLAG: --reserved-memory="" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490202 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490205 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 07 11:00:26.498044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490208 2571 flags.go:64] FLAG: --rotate-certificates="false" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490211 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490214 2571 flags.go:64] FLAG: --runonce="false" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490217 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490220 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490224 2571 flags.go:64] FLAG: --seccomp-default="false" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490229 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490231 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490235 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490238 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490241 2571 flags.go:64] FLAG: --storage-driver-password="root" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490244 2571 flags.go:64] FLAG: --storage-driver-secure="false" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490247 2571 flags.go:64] FLAG: --storage-driver-table="stats" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490250 2571 flags.go:64] FLAG: --storage-driver-user="root" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490253 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490255 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490258 2571 flags.go:64] FLAG: --system-cgroups="" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490261 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490266 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490269 2571 flags.go:64] FLAG: --tls-cert-file="" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490272 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490276 2571 flags.go:64] FLAG: --tls-min-version="" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490279 2571 flags.go:64] FLAG: --tls-private-key-file="" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490282 2571 flags.go:64] FLAG: --topology-manager-policy="none" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490284 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 07 11:00:26.498688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490287 2571 flags.go:64] FLAG: --topology-manager-scope="container" Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490290 2571 flags.go:64] FLAG: --v="2" Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490294 2571 flags.go:64] FLAG: --version="false" Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490298 2571 flags.go:64] FLAG: --vmodule="" Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490302 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.490305 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491060 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491074 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491080 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491085 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491089 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491094 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491101 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491115 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491119 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491128 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491132 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491136 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491153 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491158 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491163 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 11:00:26.499322 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491168 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491172 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491176 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491180 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491184 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491189 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491193 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491201 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491206 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491210 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491214 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491218 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491222 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491226 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491230 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491235 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491239 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491244 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491248 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491252 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 11:00:26.499848 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491261 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491265 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491270 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491276 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491282 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491286 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491290 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491294 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491299 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491303 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491308 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491312 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491321 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491325 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491329 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491333 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491337 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491343 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491349 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 11:00:26.500332 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491354 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491358 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491362 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491366 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491370 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491379 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491383 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491387 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491391 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491395 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491400 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491404 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491409 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491413 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491419 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491424 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491430 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491442 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491460 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491464 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 11:00:26.500805 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491468 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491472 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491477 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491480 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491486 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491490 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491494 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491557 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491568 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491588 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491592 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491608 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491611 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491614 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491721 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491725 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491730 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491734 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491738 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 11:00:26.501279 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491741 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491744 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491747 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491750 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491753 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.491756 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.492457 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.500301 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.500318 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.501204 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.501212 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501273 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501280 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501284 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 11:00:26.501764 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501287 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501291 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501294 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501297 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501300 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501303 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501306 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501308 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501311 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501314 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501316 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501319 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501321 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501324 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501326 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501329 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501332 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501335 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501337 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501340 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 11:00:26.502126 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501343 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501345 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501348 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501350 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501353 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501357 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501360 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501363 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501366 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501368 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501371 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501375 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501379 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501382 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501385 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501388 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501391 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501393 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501396 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 11:00:26.502640 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501399 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501402 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501404 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501407 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501410 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501412 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501415 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501417 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501420 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501422 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501425 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501428 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501430 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501433 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501436 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501438 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501441 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501455 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501461 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501464 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 11:00:26.503096 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501467 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501470 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501472 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501475 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501478 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501480 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501483 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501485 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501488 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501491 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501494 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501496 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501499 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501502 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501504 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501507 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501510 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501512 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501515 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 11:00:26.503612 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501518 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501521 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501525 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501529 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501532 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501535 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501538 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501541 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501544 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501547 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501550 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501554 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501558 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501560 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501563 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501566 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501569 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 11:00:26.504056 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501571 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.501577 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501693 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501700 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501704 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501707 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501710 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501713 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501716 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501719 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501722 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501724 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501727 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501730 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501732 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 11:00:26.504554 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501735 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501737 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501740 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501743 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501745 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501748 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501750 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501753 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501757 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501760 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501762 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501765 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501768 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501772 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501774 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501778 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501780 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501783 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501785 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501788 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 11:00:26.505033 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501790 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501793 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501795 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501798 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501800 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501803 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501805 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501807 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501810 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501813 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501815 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501818 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501820 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501823 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501826 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501828 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501831 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501833 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501836 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501838 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 11:00:26.505542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501841 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501844 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501847 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501851 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501855 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501857 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501861 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501863 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501866 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501869 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501871 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501874 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501876 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501879 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501881 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501884 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501887 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501889 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501892 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 11:00:26.506013 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501894 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501897 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501899 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501902 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501904 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501906 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501909 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501911 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501914 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501916 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501919 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501921 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501924 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501926 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501929 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501932 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501935 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501937 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501940 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 11:00:26.506475 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501942 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 11:00:26.506929 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501946 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 11:00:26.506929 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501950 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 11:00:26.506929 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501953 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 11:00:26.506929 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501956 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 11:00:26.506929 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501959 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 11:00:26.506929 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501961 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 11:00:26.506929 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:26.501964 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 11:00:26.506929 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.501969 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 11:00:26.506929 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.502115 2571 server.go:962] "Client rotation is on, will bootstrap in background" Sep 07 11:00:26.506929 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.506600 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 07 11:00:26.507706 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.507693 2571 server.go:1019] "Starting client certificate rotation" Sep 07 11:00:26.507814 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.507797 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 11:00:26.507847 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.507838 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 11:00:26.512489 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.512474 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 11:00:26.513618 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.513602 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 11:00:26.522653 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.522635 2571 log.go:25] "Validated CRI v1 runtime API" Sep 07 11:00:26.525929 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.525912 2571 log.go:25] "Validated CRI v1 image API" Sep 07 11:00:26.530620 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.530600 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 07 11:00:26.534024 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.533999 2571 fs.go:135] Filesystem UUIDs: map[039b46e1-251d-4396-bee0-775dd6d48b06:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b31dbbae-54d9-4de3-8681-cffdeaaf417b:/dev/nvme0n1p3] Sep 07 11:00:26.534118 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.534019 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 07 11:00:26.539087 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.538977 2571 manager.go:216] Machine: {Timestamp:2026-09-07 11:00:26.537646684 +0000 UTC m=+0.376104658 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200438 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27df2a265e1aebd11be1975b7ce01c SystemUUID:ec27df2a-265e-1aeb-d11b-e1975b7ce01c BootID:88774290-2d69-4bce-ba86-316a26db4e2e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0d:b2:73:13:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0d:b2:73:13:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:dd:53:62:08:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 07 11:00:26.539087 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.539081 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 07 11:00:26.539222 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.539204 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 07 11:00:26.540337 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.540319 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 11:00:26.542947 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.542910 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 07 11:00:26.543089 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.542950 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-34.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 07 11:00:26.543781 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.543771 2571 topology_manager.go:138] "Creating topology manager with none policy" Sep 07 11:00:26.543821 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.543782 2571 container_manager_linux.go:309] "Creating device plugin manager" Sep 07 11:00:26.543821 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.543790 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 11:00:26.543821 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.543803 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 11:00:26.545680 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.545671 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 07 11:00:26.550457 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.550434 2571 state_mem.go:36] "Initialized new in-memory state store" Sep 07 11:00:26.550567 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.550558 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 07 11:00:26.550605 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.550588 2571 kubelet.go:486] "Attempting to sync node with API server" Sep 07 11:00:26.550605 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.550598 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 07 11:00:26.550659 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.550615 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 07 11:00:26.550659 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.550626 2571 kubelet.go:398] "Adding apiserver pod source" Sep 07 11:00:26.550659 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.550637 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 07 11:00:26.551027 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.551010 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 11:00:26.551027 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.551030 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 11:00:26.553364 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.553346 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 07 11:00:26.556135 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556119 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 07 11:00:26.556205 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556150 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 07 11:00:26.556319 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556306 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 07 11:00:26.556319 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556320 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 07 11:00:26.556463 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556327 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 07 11:00:26.556463 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556332 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 07 11:00:26.556463 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556351 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 07 11:00:26.556463 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556357 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 07 11:00:26.556962 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556954 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 07 11:00:26.556962 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556962 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 07 11:00:26.557016 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556968 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 07 11:00:26.557016 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556975 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 07 11:00:26.557016 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556989 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 07 11:00:26.557016 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.556997 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 07 11:00:26.558937 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.558925 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 07 11:00:26.558937 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.558938 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 07 11:00:26.562806 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.562790 2571 server.go:1268] "Started kubelet" Sep 07 11:00:26.562906 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.562881 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 07 11:00:26.562962 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.562951 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 07 11:00:26.563031 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.563017 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-34.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 07 11:00:26.563649 ip-10-0-138-34 systemd[1]: Started Kubernetes Kubelet. Sep 07 11:00:26.563835 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.563808 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 07 11:00:26.563835 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.563813 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 07 11:00:26.563942 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.563812 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 07 11:00:26.564349 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.564330 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 07 11:00:26.568048 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.568028 2571 server.go:310] "Adding debug handlers to kubelet server" Sep 07 11:00:26.568652 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.568632 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 07 11:00:26.573767 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.572688 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-34.ec2.internal.18d304dd8b03681c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-34.ec2.internal,UID:ip-10-0-138-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-34.ec2.internal,},FirstTimestamp:2026-09-07 11:00:26.562758684 +0000 UTC m=+0.401216660,LastTimestamp:2026-09-07 11:00:26.562758684 +0000 UTC m=+0.401216660,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-34.ec2.internal,}" Sep 07 11:00:26.574172 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.574153 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 07 11:00:26.574819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.574803 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 07 11:00:26.574986 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.574970 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:26.575497 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575482 2571 factory.go:55] Registering systemd factory Sep 07 11:00:26.575497 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575500 2571 factory.go:223] Registration of the systemd container factory successfully Sep 07 11:00:26.575698 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575607 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 07 11:00:26.575698 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575609 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 07 11:00:26.575698 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575632 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 07 11:00:26.575813 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575728 2571 factory.go:153] Registering CRI-O factory Sep 07 11:00:26.575813 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575744 2571 factory.go:223] Registration of the crio container factory successfully Sep 07 11:00:26.575813 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575730 2571 reconstruct.go:97] "Volume reconstruction finished" Sep 07 11:00:26.575813 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575782 2571 reconciler.go:29] "Reconciler: start to sync state" Sep 07 11:00:26.575813 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575791 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 07 11:00:26.575813 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575808 2571 factory.go:103] Registering Raw factory Sep 07 11:00:26.575978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.575832 2571 manager.go:1196] Started watching for new ooms in manager Sep 07 11:00:26.576224 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.576210 2571 manager.go:349] Starting recovery of all containers Sep 07 11:00:26.579601 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.579573 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 07 11:00:26.579825 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.579804 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 07 11:00:26.585831 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.585808 2571 manager.go:354] Recovery completed Sep 07 11:00:26.590393 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.590380 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 11:00:26.592515 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.592494 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientMemory" Sep 07 11:00:26.592594 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.592536 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 11:00:26.592594 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.592550 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientPID" Sep 07 11:00:26.593201 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.593184 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 07 11:00:26.593201 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.593200 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 07 11:00:26.593331 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.593204 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 11:00:26.593331 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.593219 2571 state_mem.go:36] "Initialized new in-memory state store" Sep 07 11:00:26.594353 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.594279 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-34.ec2.internal.18d304dd8cc9774f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-34.ec2.internal,UID:ip-10-0-138-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-34.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-34.ec2.internal,},FirstTimestamp:2026-09-07 11:00:26.592515919 +0000 UTC m=+0.430973901,LastTimestamp:2026-09-07 11:00:26.592515919 +0000 UTC m=+0.430973901,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-34.ec2.internal,}" Sep 07 11:00:26.595406 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.595394 2571 policy_none.go:49] "None policy: Start" Sep 07 11:00:26.595581 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.595572 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 07 11:00:26.595618 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.595586 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 07 11:00:26.595656 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.595632 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dz45c" Sep 07 11:00:26.595886 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.595733 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientMemory" Sep 07 11:00:26.595886 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.595760 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 11:00:26.595886 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.595775 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientPID" Sep 07 11:00:26.597626 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.597610 2571 policy_none.go:47] "Start" Sep 07 11:00:26.604382 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.604363 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dz45c" Sep 07 11:00:26.605965 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.605897 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-34.ec2.internal.18d304dd8cc9e56f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-34.ec2.internal,UID:ip-10-0-138-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-34.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-34.ec2.internal,},FirstTimestamp:2026-09-07 11:00:26.592544111 +0000 UTC m=+0.431002086,LastTimestamp:2026-09-07 11:00:26.592544111 +0000 UTC m=+0.431002086,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-34.ec2.internal,}" Sep 07 11:00:26.609611 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.609532 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-34.ec2.internal.18d304dd8cca0da2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-34.ec2.internal,UID:ip-10-0-138-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-34.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-34.ec2.internal,},FirstTimestamp:2026-09-07 11:00:26.592554402 +0000 UTC m=+0.431012376,LastTimestamp:2026-09-07 11:00:26.592554402 +0000 UTC m=+0.431012376,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-34.ec2.internal,}" Sep 07 11:00:26.627193 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.627173 2571 manager.go:337] "Starting Device Plugin manager" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.627224 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.627236 2571 server.go:87] "Starting device plugin registration server" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.627531 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.627543 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.627690 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.627763 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.627772 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.628246 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.628288 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.644079 2571 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2605 into the system container \"/system.slice\": " Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.728308 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.729211 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientMemory" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.729243 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.729258 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientPID" Sep 07 11:00:26.739723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.729290 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.740195 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.740168 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.740229 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.740201 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-34.ec2.internal\": node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:26.752501 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.752471 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 07 11:00:26.753709 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.753692 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 07 11:00:26.753821 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.753745 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 07 11:00:26.753821 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.753773 2571 kubelet.go:2448] "Starting kubelet main sync loop" Sep 07 11:00:26.753914 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.753818 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 07 11:00:26.755696 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.755670 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 11:00:26.763743 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.763714 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:26.854434 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.854397 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-34.ec2.internal"] Sep 07 11:00:26.854591 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.854512 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 11:00:26.856146 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.856129 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientMemory" Sep 07 11:00:26.856244 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.856157 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 11:00:26.856244 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.856167 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientPID" Sep 07 11:00:26.858545 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.858533 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 11:00:26.858669 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.858647 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.858707 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.858689 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 11:00:26.859242 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.859225 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientMemory" Sep 07 11:00:26.859242 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.859242 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientMemory" Sep 07 11:00:26.859396 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.859256 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 11:00:26.859396 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.859266 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 11:00:26.859396 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.859269 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientPID" Sep 07 11:00:26.859396 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.859276 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientPID" Sep 07 11:00:26.861524 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.861508 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.861616 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.861538 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 11:00:26.862270 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.862255 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientMemory" Sep 07 11:00:26.862401 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.862281 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 11:00:26.862401 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.862293 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeHasSufficientPID" Sep 07 11:00:26.863947 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.863928 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:26.877525 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.877489 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/57f2b351dd062a0c56b2c5fe55d5b879-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal\" (UID: \"57f2b351dd062a0c56b2c5fe55d5b879\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.877525 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.877520 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57f2b351dd062a0c56b2c5fe55d5b879-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal\" (UID: \"57f2b351dd062a0c56b2c5fe55d5b879\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.877653 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.877540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/77bb3ed25ee7e95473707df13e75eca1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-34.ec2.internal\" (UID: \"77bb3ed25ee7e95473707df13e75eca1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.893487 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.893466 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-34.ec2.internal\" not found" node="ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.898104 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.898088 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-34.ec2.internal\" not found" node="ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.964203 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:26.964160 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:26.978569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.978541 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/57f2b351dd062a0c56b2c5fe55d5b879-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal\" (UID: \"57f2b351dd062a0c56b2c5fe55d5b879\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.978679 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.978574 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57f2b351dd062a0c56b2c5fe55d5b879-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal\" (UID: \"57f2b351dd062a0c56b2c5fe55d5b879\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.978679 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.978589 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/77bb3ed25ee7e95473707df13e75eca1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-34.ec2.internal\" (UID: \"77bb3ed25ee7e95473707df13e75eca1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.978679 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.978656 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/77bb3ed25ee7e95473707df13e75eca1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-34.ec2.internal\" (UID: \"77bb3ed25ee7e95473707df13e75eca1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.978679 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.978657 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/57f2b351dd062a0c56b2c5fe55d5b879-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal\" (UID: \"57f2b351dd062a0c56b2c5fe55d5b879\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" Sep 07 11:00:26.978815 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:26.978655 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57f2b351dd062a0c56b2c5fe55d5b879-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal\" (UID: \"57f2b351dd062a0c56b2c5fe55d5b879\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" Sep 07 11:00:27.065045 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:27.064972 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:27.165576 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:27.165549 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:27.197020 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:27.196987 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" Sep 07 11:00:27.202319 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:27.202160 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.ec2.internal" Sep 07 11:00:27.266617 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:27.266430 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:28.731115 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:28.731092 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:28.831740 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:28.831699 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:28.871330 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:28.871296 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 07 11:00:28.871796 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:28.871502 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 07 11:00:28.932433 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:28.932404 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:28.971920 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:28.971877 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 10:55:27 +0000 UTC" deadline="2028-05-22 21:32:02.689802326 +0000 UTC" Sep 07 11:00:28.971920 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:28.971910 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14962h31m33.717895377s" Sep 07 11:00:29.033211 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:29.033142 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-34.ec2.internal\" not found" Sep 07 11:00:29.064836 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.064802 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 11:00:29.139612 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.139585 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" Sep 07 11:00:29.150600 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.150577 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 11:00:29.154034 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.154016 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 11:00:29.155132 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.155118 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.ec2.internal" Sep 07 11:00:29.163534 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.163506 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 11:00:29.207992 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.207963 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 11:00:29.371704 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:29.371655 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f2b351dd062a0c56b2c5fe55d5b879.slice/crio-f26edac68c185c61f34d71e270746de73c378f8ef3c89e5516ee1d60682c2fdf WatchSource:0}: Error finding container f26edac68c185c61f34d71e270746de73c378f8ef3c89e5516ee1d60682c2fdf: Status 404 returned error can't find the container with id f26edac68c185c61f34d71e270746de73c378f8ef3c89e5516ee1d60682c2fdf Sep 07 11:00:29.372321 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:29.372298 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bb3ed25ee7e95473707df13e75eca1.slice/crio-374662f6c2bd3db08aa091c3f960974d46ba29cd51df84aab95a616397f37deb WatchSource:0}: Error finding container 374662f6c2bd3db08aa091c3f960974d46ba29cd51df84aab95a616397f37deb: Status 404 returned error can't find the container with id 374662f6c2bd3db08aa091c3f960974d46ba29cd51df84aab95a616397f37deb Sep 07 11:00:29.376584 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.376568 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 11:00:29.916013 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.915973 2571 apiserver.go:52] "Watching apiserver" Sep 07 11:00:29.923498 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.923476 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 07 11:00:29.923863 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.923840 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-34.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77","openshift-cluster-node-tuning-operator/tuned-zh2px","openshift-dns/node-resolver-4fw4g","openshift-image-registry/node-ca-r5q7t","openshift-multus/multus-m6xpc","openshift-multus/network-metrics-daemon-6jc59","openshift-network-operator/iptables-alerter-zqj2l","kube-system/global-pull-secret-syncer-4vbhj","kube-system/konnectivity-agent-p2db7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal","openshift-multus/multus-additional-cni-plugins-nv7jl","openshift-network-diagnostics/network-check-target-qz5nc","openshift-ovn-kubernetes/ovnkube-node-rlg9h"] Sep 07 11:00:29.927645 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.927623 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:29.927750 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:29.927717 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:29.929621 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.929607 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:29.929678 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:29.929662 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:29.931756 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.931744 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.933800 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.933780 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 07 11:00:29.933910 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.933894 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4fw4g" Sep 07 11:00:29.934038 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.934021 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z2vbz\"" Sep 07 11:00:29.934114 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.934036 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 07 11:00:29.935989 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.935972 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r5q7t" Sep 07 11:00:29.936697 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.936674 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 07 11:00:29.936793 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.936761 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 07 11:00:29.936842 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.936816 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qhknf\"" Sep 07 11:00:29.938124 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.937981 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.938183 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.938141 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 07 11:00:29.938558 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.938528 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 07 11:00:29.938663 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.938534 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 07 11:00:29.938663 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.938633 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q2mvz\"" Sep 07 11:00:29.938771 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.938678 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 07 11:00:29.940455 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.940245 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:29.940455 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.940321 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 07 11:00:29.940455 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:29.940334 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:29.940455 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.940350 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 07 11:00:29.940455 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.940356 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 07 11:00:29.940744 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.940722 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5t9zd\"" Sep 07 11:00:29.940780 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.940748 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 07 11:00:29.942620 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.942596 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:29.945309 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.945294 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:29.946413 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.946022 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 07 11:00:29.946413 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.946081 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 07 11:00:29.946413 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.946094 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 07 11:00:29.946413 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.946129 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 07 11:00:29.946413 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.946190 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f64f5\"" Sep 07 11:00:29.946413 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.946087 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 07 11:00:29.946413 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.946251 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 07 11:00:29.947534 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.947515 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4tf5g\"" Sep 07 11:00:29.947534 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.947529 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 07 11:00:29.947703 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.947612 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:29.948906 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.948690 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 07 11:00:29.948906 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.948847 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 07 11:00:29.950452 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.949212 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 11:00:29.950534 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.950436 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nmpzr\"" Sep 07 11:00:29.950614 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.950596 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 07 11:00:29.950731 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.950714 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 07 11:00:29.950887 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.950870 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-sysconfig\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.950955 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.950912 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx5s2\" (UniqueName: \"kubernetes.io/projected/d4897f70-7801-4ebe-a9ff-b1df36797e92-kube-api-access-nx5s2\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.951012 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.950970 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c839fa80-c4fe-4444-b4d3-4de7cef50ec6-tmp-dir\") pod \"node-resolver-4fw4g\" (UID: \"c839fa80-c4fe-4444-b4d3-4de7cef50ec6\") " pod="openshift-dns/node-resolver-4fw4g" Sep 07 11:00:29.951012 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.950992 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-cnibin\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.951107 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951037 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-var-lib-kubelet\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.951107 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951061 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-multus-conf-dir\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.951107 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951093 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-system-cni-dir\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.951378 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951126 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-etc-kubernetes\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.951378 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951154 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/973302dc-6097-4017-b889-c7bcd790d603-host\") pod \"node-ca-r5q7t\" (UID: \"973302dc-6097-4017-b889-c7bcd790d603\") " pod="openshift-image-registry/node-ca-r5q7t" Sep 07 11:00:29.951378 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951180 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17610838-7a01-4354-970a-3aa13a7e64f9-cni-binary-copy\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.951378 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951226 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-run-k8s-cni-cncf-io\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.951378 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951270 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-kubernetes\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.951378 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951303 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-systemd\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.951648 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951398 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-host\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.951648 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951428 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnj2\" (UniqueName: \"kubernetes.io/projected/c839fa80-c4fe-4444-b4d3-4de7cef50ec6-kube-api-access-clnj2\") pod \"node-resolver-4fw4g\" (UID: \"c839fa80-c4fe-4444-b4d3-4de7cef50ec6\") " pod="openshift-dns/node-resolver-4fw4g" Sep 07 11:00:29.951648 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951460 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/973302dc-6097-4017-b889-c7bcd790d603-serviceca\") pod \"node-ca-r5q7t\" (UID: \"973302dc-6097-4017-b889-c7bcd790d603\") " pod="openshift-image-registry/node-ca-r5q7t" Sep 07 11:00:29.951648 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951489 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-multus-cni-dir\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.951894 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951868 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-kubelet-config\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:29.951992 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951915 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zqj2l" Sep 07 11:00:29.952050 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.951915 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-modprobe-d\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.952050 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952035 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4897f70-7801-4ebe-a9ff-b1df36797e92-tmp\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.952147 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952063 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-hostroot\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.952147 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952088 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-sysctl-d\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.952147 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952125 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-sysctl-conf\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.952279 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952167 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-run\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.952279 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952208 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c839fa80-c4fe-4444-b4d3-4de7cef50ec6-hosts-file\") pod \"node-resolver-4fw4g\" (UID: \"c839fa80-c4fe-4444-b4d3-4de7cef50ec6\") " pod="openshift-dns/node-resolver-4fw4g" Sep 07 11:00:29.952279 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952238 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-dbus\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:29.952279 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952257 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8zw\" (UniqueName: \"kubernetes.io/projected/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-kube-api-access-7z8zw\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:29.952279 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952280 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n554\" (UniqueName: \"kubernetes.io/projected/973302dc-6097-4017-b889-c7bcd790d603-kube-api-access-8n554\") pod \"node-ca-r5q7t\" (UID: \"973302dc-6097-4017-b889-c7bcd790d603\") " pod="openshift-image-registry/node-ca-r5q7t" Sep 07 11:00:29.952480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952294 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-multus-socket-dir-parent\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.952480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952307 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/17610838-7a01-4354-970a-3aa13a7e64f9-multus-daemon-config\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.952480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952336 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-run-multus-certs\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.952480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952365 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-var-lib-kubelet\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.952480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952404 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-tuned\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.952480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952432 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:29.952480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952448 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwrm\" (UniqueName: \"kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm\") pod \"network-check-target-qz5nc\" (UID: \"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084\") " pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:29.952480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952470 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-var-lib-cni-bin\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.952757 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952501 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-var-lib-cni-multus\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.952757 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952517 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:29.952757 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952530 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-sys\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.952757 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952542 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-lib-modules\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:29.952757 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952594 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-run-netns\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.952757 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.952624 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5kx\" (UniqueName: \"kubernetes.io/projected/17610838-7a01-4354-970a-3aa13a7e64f9-kube-api-access-gw5kx\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:29.954357 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.954311 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 07 11:00:29.954464 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.954396 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 07 11:00:29.954464 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.954397 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x5k44\"" Sep 07 11:00:29.954464 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.954456 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:29.954640 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.954530 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 07 11:00:29.957253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.957161 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8h6xz\"" Sep 07 11:00:29.957253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.957207 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 07 11:00:29.957349 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.957254 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 07 11:00:29.969049 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.969034 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rtb96" Sep 07 11:00:29.977143 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:29.977124 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rtb96" Sep 07 11:00:30.041290 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.041260 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 07 11:00:30.053765 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.053733 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-cnibin\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.053765 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.053767 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbdbz\" (UniqueName: \"kubernetes.io/projected/9b81c80c-61cb-43ce-ba52-11ba84bc5a12-kube-api-access-mbdbz\") pod \"iptables-alerter-zqj2l\" (UID: \"9b81c80c-61cb-43ce-ba52-11ba84bc5a12\") " pod="openshift-network-operator/iptables-alerter-zqj2l" Sep 07 11:00:30.054000 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.053795 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-etc-kubernetes\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.054000 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.053858 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-etc-kubernetes\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.054000 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.053887 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b81c80c-61cb-43ce-ba52-11ba84bc5a12-iptables-alerter-script\") pod \"iptables-alerter-zqj2l\" (UID: \"9b81c80c-61cb-43ce-ba52-11ba84bc5a12\") " pod="openshift-network-operator/iptables-alerter-zqj2l" Sep 07 11:00:30.054000 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.053906 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-cnibin\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.054000 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.053922 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-run-ovn\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.054000 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.053950 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-kubernetes\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.054269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054037 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-host\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.054269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054043 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-kubernetes\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.054269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054064 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/973302dc-6097-4017-b889-c7bcd790d603-serviceca\") pod \"node-ca-r5q7t\" (UID: \"973302dc-6097-4017-b889-c7bcd790d603\") " pod="openshift-image-registry/node-ca-r5q7t" Sep 07 11:00:30.054269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054081 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-host\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.054269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054088 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-modprobe-d\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.054269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054140 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4897f70-7801-4ebe-a9ff-b1df36797e92-tmp\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.054269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054170 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-node-log\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.054269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054196 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.054269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054224 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-modprobe-d\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.054269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054226 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36904d67-6a0b-4530-bfda-f87040d0dd71-konnectivity-ca\") pod \"konnectivity-agent-p2db7\" (UID: \"36904d67-6a0b-4530-bfda-f87040d0dd71\") " pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:30.054683 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054265 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-sysctl-d\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.054683 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054303 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-run-systemd\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.054683 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054346 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/484ea397-1fd4-44c3-8295-9c1833163ccc-ovn-node-metrics-cert\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.054683 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054378 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.054683 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054409 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-sysctl-d\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.054683 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054448 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-kubelet\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.054683 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054520 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 07 11:00:30.054928 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054853 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8zw\" (UniqueName: \"kubernetes.io/projected/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-kube-api-access-7z8zw\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:30.054978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054928 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-multus-socket-dir-parent\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.055028 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.054991 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-run-multus-certs\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.055080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055031 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-var-lib-kubelet\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.055080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055041 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-multus-socket-dir-parent\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.055080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055057 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-log-socket\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.055213 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055084 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-run-multus-certs\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.055213 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055081 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffw9d\" (UniqueName: \"kubernetes.io/projected/484ea397-1fd4-44c3-8295-9c1833163ccc-kube-api-access-ffw9d\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.055213 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055114 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-socket-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.055213 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055170 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-var-lib-kubelet\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.055213 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055169 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwrm\" (UniqueName: \"kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm\") pod \"network-check-target-qz5nc\" (UID: \"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084\") " pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:30.055414 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055274 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/973302dc-6097-4017-b889-c7bcd790d603-serviceca\") pod \"node-ca-r5q7t\" (UID: \"973302dc-6097-4017-b889-c7bcd790d603\") " pod="openshift-image-registry/node-ca-r5q7t" Sep 07 11:00:30.055414 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055290 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-var-lib-cni-multus\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.055414 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055331 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:30.055414 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055334 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-var-lib-cni-multus\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.055414 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055364 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-lib-modules\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.055414 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055387 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-run-ovn-kubernetes\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.055417 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055425 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055449 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-run-netns\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055464 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-lib-modules\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.055475 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret podName:7fbcd4d6-16cc-4ba2-933f-6938932d59a3 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:30.555457477 +0000 UTC m=+3.029813833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret") pod "global-pull-secret-syncer-4vbhj" (UID: "7fbcd4d6-16cc-4ba2-933f-6938932d59a3") : object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055481 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-run-netns\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055505 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055543 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-var-lib-kubelet\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055580 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-multus-conf-dir\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055601 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-slash\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055630 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-run-openvswitch\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055659 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-multus-conf-dir\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055654 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.055695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055696 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-system-cni-dir\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055701 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-var-lib-kubelet\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055713 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-system-cni-dir\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055730 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/973302dc-6097-4017-b889-c7bcd790d603-host\") pod \"node-ca-r5q7t\" (UID: \"973302dc-6097-4017-b889-c7bcd790d603\") " pod="openshift-image-registry/node-ca-r5q7t" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055744 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-system-cni-dir\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055745 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17610838-7a01-4354-970a-3aa13a7e64f9-cni-binary-copy\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055776 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-run-k8s-cni-cncf-io\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055797 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/973302dc-6097-4017-b889-c7bcd790d603-host\") pod \"node-ca-r5q7t\" (UID: \"973302dc-6097-4017-b889-c7bcd790d603\") " pod="openshift-image-registry/node-ca-r5q7t" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-systemd\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055833 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnj2\" (UniqueName: \"kubernetes.io/projected/c839fa80-c4fe-4444-b4d3-4de7cef50ec6-kube-api-access-clnj2\") pod \"node-resolver-4fw4g\" (UID: \"c839fa80-c4fe-4444-b4d3-4de7cef50ec6\") " pod="openshift-dns/node-resolver-4fw4g" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055842 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-systemd\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055807 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-run-k8s-cni-cncf-io\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055860 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-systemd-units\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055882 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-cni-netd\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055910 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-multus-cni-dir\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055936 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-kubelet-config\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.055962 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-var-lib-openvswitch\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.056253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056006 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-cni-bin\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056028 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056031 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-multus-cni-dir\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056021 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-kubelet-config\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056053 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/484ea397-1fd4-44c3-8295-9c1833163ccc-ovnkube-config\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056076 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-sys-fs\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056129 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36904d67-6a0b-4530-bfda-f87040d0dd71-agent-certs\") pod \"konnectivity-agent-p2db7\" (UID: \"36904d67-6a0b-4530-bfda-f87040d0dd71\") " pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056150 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-hostroot\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056170 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-sysctl-conf\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-run\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056201 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17610838-7a01-4354-970a-3aa13a7e64f9-cni-binary-copy\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056212 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c839fa80-c4fe-4444-b4d3-4de7cef50ec6-hosts-file\") pod \"node-resolver-4fw4g\" (UID: \"c839fa80-c4fe-4444-b4d3-4de7cef50ec6\") " pod="openshift-dns/node-resolver-4fw4g" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056220 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-hostroot\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056233 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-run-netns\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056256 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knfl4\" (UniqueName: \"kubernetes.io/projected/522d0108-c655-4fee-b481-82f8570276b2-kube-api-access-knfl4\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056278 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-etc-openvswitch\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056281 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c839fa80-c4fe-4444-b4d3-4de7cef50ec6-hosts-file\") pod \"node-resolver-4fw4g\" (UID: \"c839fa80-c4fe-4444-b4d3-4de7cef50ec6\") " pod="openshift-dns/node-resolver-4fw4g" Sep 07 11:00:30.057026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056307 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-dbus\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056272 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-run\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n554\" (UniqueName: \"kubernetes.io/projected/973302dc-6097-4017-b889-c7bcd790d603-kube-api-access-8n554\") pod \"node-ca-r5q7t\" (UID: \"973302dc-6097-4017-b889-c7bcd790d603\") " pod="openshift-image-registry/node-ca-r5q7t" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056328 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-sysctl-conf\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056372 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-dbus\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056384 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/17610838-7a01-4354-970a-3aa13a7e64f9-multus-daemon-config\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056468 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-tuned\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056499 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/484ea397-1fd4-44c3-8295-9c1833163ccc-env-overrides\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056536 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056618 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4gjg\" (UniqueName: \"kubernetes.io/projected/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-kube-api-access-m4gjg\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056653 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056676 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-var-lib-cni-bin\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056715 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-sys\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056775 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx5s2\" (UniqueName: \"kubernetes.io/projected/d4897f70-7801-4ebe-a9ff-b1df36797e92-kube-api-access-nx5s2\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056804 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c839fa80-c4fe-4444-b4d3-4de7cef50ec6-tmp-dir\") pod \"node-resolver-4fw4g\" (UID: \"c839fa80-c4fe-4444-b4d3-4de7cef50ec6\") " pod="openshift-dns/node-resolver-4fw4g" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056865 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-sys\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.056868 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:30.057724 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.056940 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs podName:cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:30.556922736 +0000 UTC m=+3.031279098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs") pod "network-metrics-daemon-6jc59" (UID: "cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:30.058327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.056994 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17610838-7a01-4354-970a-3aa13a7e64f9-host-var-lib-cni-bin\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.058327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.057007 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/17610838-7a01-4354-970a-3aa13a7e64f9-multus-daemon-config\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.058327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.057021 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b81c80c-61cb-43ce-ba52-11ba84bc5a12-host-slash\") pod \"iptables-alerter-zqj2l\" (UID: \"9b81c80c-61cb-43ce-ba52-11ba84bc5a12\") " pod="openshift-network-operator/iptables-alerter-zqj2l" Sep 07 11:00:30.058327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.057054 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-registration-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.058327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.057090 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5kx\" (UniqueName: \"kubernetes.io/projected/17610838-7a01-4354-970a-3aa13a7e64f9-kube-api-access-gw5kx\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.058327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.057099 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c839fa80-c4fe-4444-b4d3-4de7cef50ec6-tmp-dir\") pod \"node-resolver-4fw4g\" (UID: \"c839fa80-c4fe-4444-b4d3-4de7cef50ec6\") " pod="openshift-dns/node-resolver-4fw4g" Sep 07 11:00:30.058327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.057170 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-sysconfig\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.058327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.057216 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/484ea397-1fd4-44c3-8295-9c1833163ccc-ovnkube-script-lib\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.058327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.057239 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-sysconfig\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.058327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.057281 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-device-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.058327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.057324 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-cnibin\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.058780 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.058460 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4897f70-7801-4ebe-a9ff-b1df36797e92-tmp\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.058780 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.058723 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4897f70-7801-4ebe-a9ff-b1df36797e92-etc-tuned\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.062459 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.062440 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 11:00:30.062613 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.062464 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 11:00:30.062613 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.062477 2571 projected.go:196] Error preparing data for projected volume kube-api-access-gzwrm for pod openshift-network-diagnostics/network-check-target-qz5nc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:30.062718 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.062621 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm podName:fc08ced6-7a36-4ef2-b8e2-7cd3eb377084 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:30.562599118 +0000 UTC m=+3.036955491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwrm" (UniqueName: "kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm") pod "network-check-target-qz5nc" (UID: "fc08ced6-7a36-4ef2-b8e2-7cd3eb377084") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:30.064467 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.064437 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8zw\" (UniqueName: \"kubernetes.io/projected/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-kube-api-access-7z8zw\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:30.065010 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.064983 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnj2\" (UniqueName: \"kubernetes.io/projected/c839fa80-c4fe-4444-b4d3-4de7cef50ec6-kube-api-access-clnj2\") pod \"node-resolver-4fw4g\" (UID: \"c839fa80-c4fe-4444-b4d3-4de7cef50ec6\") " pod="openshift-dns/node-resolver-4fw4g" Sep 07 11:00:30.065665 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.065624 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n554\" (UniqueName: \"kubernetes.io/projected/973302dc-6097-4017-b889-c7bcd790d603-kube-api-access-8n554\") pod \"node-ca-r5q7t\" (UID: \"973302dc-6097-4017-b889-c7bcd790d603\") " pod="openshift-image-registry/node-ca-r5q7t" Sep 07 11:00:30.065665 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.065656 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5kx\" (UniqueName: \"kubernetes.io/projected/17610838-7a01-4354-970a-3aa13a7e64f9-kube-api-access-gw5kx\") pod \"multus-m6xpc\" (UID: \"17610838-7a01-4354-970a-3aa13a7e64f9\") " pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.066830 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.066808 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx5s2\" (UniqueName: \"kubernetes.io/projected/d4897f70-7801-4ebe-a9ff-b1df36797e92-kube-api-access-nx5s2\") pod \"tuned-zh2px\" (UID: \"d4897f70-7801-4ebe-a9ff-b1df36797e92\") " pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.119646 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.119615 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 11:00:30.122373 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.122318 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.ec2.internal" event={"ID":"77bb3ed25ee7e95473707df13e75eca1","Type":"ContainerStarted","Data":"374662f6c2bd3db08aa091c3f960974d46ba29cd51df84aab95a616397f37deb"} Sep 07 11:00:30.123627 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.123573 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" event={"ID":"57f2b351dd062a0c56b2c5fe55d5b879","Type":"ContainerStarted","Data":"f26edac68c185c61f34d71e270746de73c378f8ef3c89e5516ee1d60682c2fdf"} Sep 07 11:00:30.157840 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.157806 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-node-log\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.157840 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.157843 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.158074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.157859 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36904d67-6a0b-4530-bfda-f87040d0dd71-konnectivity-ca\") pod \"konnectivity-agent-p2db7\" (UID: \"36904d67-6a0b-4530-bfda-f87040d0dd71\") " pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:30.158074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.157882 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-run-systemd\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.157908 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/484ea397-1fd4-44c3-8295-9c1833163ccc-ovn-node-metrics-cert\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.157926 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-node-log\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.157933 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.158074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.157985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-kubelet\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158011 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-log-socket\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158014 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.158074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158037 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffw9d\" (UniqueName: \"kubernetes.io/projected/484ea397-1fd4-44c3-8295-9c1833163ccc-kube-api-access-ffw9d\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158061 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-socket-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.158074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158052 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-run-systemd\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158069 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-kubelet\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158106 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-log-socket\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158122 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-run-ovn-kubernetes\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158152 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158184 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158194 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-socket-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158211 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-slash\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158235 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-run-openvswitch\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158261 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158286 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-system-cni-dir\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158316 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-systemd-units\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158338 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-cni-netd\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158375 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-var-lib-openvswitch\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158386 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-run-openvswitch\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158399 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-cni-bin\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158434 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.158655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158450 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-run-ovn-kubernetes\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158457 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/484ea397-1fd4-44c3-8295-9c1833163ccc-ovnkube-config\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158493 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-sys-fs\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158510 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36904d67-6a0b-4530-bfda-f87040d0dd71-agent-certs\") pod \"konnectivity-agent-p2db7\" (UID: \"36904d67-6a0b-4530-bfda-f87040d0dd71\") " pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158519 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158529 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-run-netns\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158518 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36904d67-6a0b-4530-bfda-f87040d0dd71-konnectivity-ca\") pod \"konnectivity-agent-p2db7\" (UID: \"36904d67-6a0b-4530-bfda-f87040d0dd71\") " pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158563 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knfl4\" (UniqueName: \"kubernetes.io/projected/522d0108-c655-4fee-b481-82f8570276b2-kube-api-access-knfl4\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158585 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-sys-fs\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158603 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-etc-openvswitch\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158632 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-slash\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158635 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/484ea397-1fd4-44c3-8295-9c1833163ccc-env-overrides\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158662 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158663 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158669 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-etc-openvswitch\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158701 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gjg\" (UniqueName: \"kubernetes.io/projected/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-kube-api-access-m4gjg\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158747 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b81c80c-61cb-43ce-ba52-11ba84bc5a12-host-slash\") pod \"iptables-alerter-zqj2l\" (UID: \"9b81c80c-61cb-43ce-ba52-11ba84bc5a12\") " pod="openshift-network-operator/iptables-alerter-zqj2l" Sep 07 11:00:30.159422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158750 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-run-netns\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158776 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-cni-netd\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158797 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-registration-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158805 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158815 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/484ea397-1fd4-44c3-8295-9c1833163ccc-ovnkube-script-lib\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158857 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158865 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158883 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-system-cni-dir\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158909 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-systemd-units\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158923 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-cni-bin\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158935 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-registration-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158961 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b81c80c-61cb-43ce-ba52-11ba84bc5a12-host-slash\") pod \"iptables-alerter-zqj2l\" (UID: \"9b81c80c-61cb-43ce-ba52-11ba84bc5a12\") " pod="openshift-network-operator/iptables-alerter-zqj2l" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158964 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-var-lib-openvswitch\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158979 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-device-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.158988 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.159033 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-cnibin\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.159042 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/522d0108-c655-4fee-b481-82f8570276b2-device-dir\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.160221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.159059 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbdbz\" (UniqueName: \"kubernetes.io/projected/9b81c80c-61cb-43ce-ba52-11ba84bc5a12-kube-api-access-mbdbz\") pod \"iptables-alerter-zqj2l\" (UID: \"9b81c80c-61cb-43ce-ba52-11ba84bc5a12\") " pod="openshift-network-operator/iptables-alerter-zqj2l" Sep 07 11:00:30.161014 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.159069 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/484ea397-1fd4-44c3-8295-9c1833163ccc-ovnkube-config\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.161014 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.159079 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-cnibin\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.161014 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.159082 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b81c80c-61cb-43ce-ba52-11ba84bc5a12-iptables-alerter-script\") pod \"iptables-alerter-zqj2l\" (UID: \"9b81c80c-61cb-43ce-ba52-11ba84bc5a12\") " pod="openshift-network-operator/iptables-alerter-zqj2l" Sep 07 11:00:30.161014 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.159115 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-run-ovn\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.161014 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.159118 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/484ea397-1fd4-44c3-8295-9c1833163ccc-env-overrides\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.161014 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.159155 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/484ea397-1fd4-44c3-8295-9c1833163ccc-run-ovn\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.161014 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.159439 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/484ea397-1fd4-44c3-8295-9c1833163ccc-ovnkube-script-lib\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.161014 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.159624 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b81c80c-61cb-43ce-ba52-11ba84bc5a12-iptables-alerter-script\") pod \"iptables-alerter-zqj2l\" (UID: \"9b81c80c-61cb-43ce-ba52-11ba84bc5a12\") " pod="openshift-network-operator/iptables-alerter-zqj2l" Sep 07 11:00:30.161351 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.161295 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36904d67-6a0b-4530-bfda-f87040d0dd71-agent-certs\") pod \"konnectivity-agent-p2db7\" (UID: \"36904d67-6a0b-4530-bfda-f87040d0dd71\") " pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:30.161575 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.161537 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/484ea397-1fd4-44c3-8295-9c1833163ccc-ovn-node-metrics-cert\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.167252 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.167188 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffw9d\" (UniqueName: \"kubernetes.io/projected/484ea397-1fd4-44c3-8295-9c1833163ccc-kube-api-access-ffw9d\") pod \"ovnkube-node-rlg9h\" (UID: \"484ea397-1fd4-44c3-8295-9c1833163ccc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.167252 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.167240 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knfl4\" (UniqueName: \"kubernetes.io/projected/522d0108-c655-4fee-b481-82f8570276b2-kube-api-access-knfl4\") pod \"aws-ebs-csi-driver-node-4qs77\" (UID: \"522d0108-c655-4fee-b481-82f8570276b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.167775 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.167747 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gjg\" (UniqueName: \"kubernetes.io/projected/54fc1277-37e8-4db4-941b-5e6c19a9a3c4-kube-api-access-m4gjg\") pod \"multus-additional-cni-plugins-nv7jl\" (UID: \"54fc1277-37e8-4db4-941b-5e6c19a9a3c4\") " pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.167912 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.167893 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbdbz\" (UniqueName: \"kubernetes.io/projected/9b81c80c-61cb-43ce-ba52-11ba84bc5a12-kube-api-access-mbdbz\") pod \"iptables-alerter-zqj2l\" (UID: \"9b81c80c-61cb-43ce-ba52-11ba84bc5a12\") " pod="openshift-network-operator/iptables-alerter-zqj2l" Sep 07 11:00:30.240810 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.240775 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zh2px" Sep 07 11:00:30.246668 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.246642 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4fw4g" Sep 07 11:00:30.250651 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:30.250619 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4897f70_7801_4ebe_a9ff_b1df36797e92.slice/crio-620ca35a642afd7d97d16cb2d16bd53cd3b8e0edf823afb8674f7ecff869abcb WatchSource:0}: Error finding container 620ca35a642afd7d97d16cb2d16bd53cd3b8e0edf823afb8674f7ecff869abcb: Status 404 returned error can't find the container with id 620ca35a642afd7d97d16cb2d16bd53cd3b8e0edf823afb8674f7ecff869abcb Sep 07 11:00:30.256145 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.255952 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r5q7t" Sep 07 11:00:30.256411 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:30.256389 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc839fa80_c4fe_4444_b4d3_4de7cef50ec6.slice/crio-6e2ed349418eb0949df322be5b34aa1f759a42a05b988922cfae112d8261fb50 WatchSource:0}: Error finding container 6e2ed349418eb0949df322be5b34aa1f759a42a05b988922cfae112d8261fb50: Status 404 returned error can't find the container with id 6e2ed349418eb0949df322be5b34aa1f759a42a05b988922cfae112d8261fb50 Sep 07 11:00:30.259648 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.259624 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m6xpc" Sep 07 11:00:30.265960 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.265940 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:30.268542 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:30.268517 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973302dc_6097_4017_b889_c7bcd790d603.slice/crio-adbc94f565edf89af211ecf2c3b20d3133e740936d05b80762d3452892c75e75 WatchSource:0}: Error finding container adbc94f565edf89af211ecf2c3b20d3133e740936d05b80762d3452892c75e75: Status 404 returned error can't find the container with id adbc94f565edf89af211ecf2c3b20d3133e740936d05b80762d3452892c75e75 Sep 07 11:00:30.269040 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:30.269020 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17610838_7a01_4354_970a_3aa13a7e64f9.slice/crio-d4de4b6d3043c844c3a4083ee5aafc3f292d6a75f954880e07c418bcad52d1c6 WatchSource:0}: Error finding container d4de4b6d3043c844c3a4083ee5aafc3f292d6a75f954880e07c418bcad52d1c6: Status 404 returned error can't find the container with id d4de4b6d3043c844c3a4083ee5aafc3f292d6a75f954880e07c418bcad52d1c6 Sep 07 11:00:30.270465 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.270437 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" Sep 07 11:00:30.276749 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.276676 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:30.278035 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:30.278011 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484ea397_1fd4_44c3_8295_9c1833163ccc.slice/crio-94021bd430680b3a901ca23c787d07f784f357adb966eebeaacd1b01e7081fbc WatchSource:0}: Error finding container 94021bd430680b3a901ca23c787d07f784f357adb966eebeaacd1b01e7081fbc: Status 404 returned error can't find the container with id 94021bd430680b3a901ca23c787d07f784f357adb966eebeaacd1b01e7081fbc Sep 07 11:00:30.281900 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:30.281868 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522d0108_c655_4fee_b481_82f8570276b2.slice/crio-58237ca26e5e966952cffb95345ac75b84263006b9b0eec8d5aa556931942589 WatchSource:0}: Error finding container 58237ca26e5e966952cffb95345ac75b84263006b9b0eec8d5aa556931942589: Status 404 returned error can't find the container with id 58237ca26e5e966952cffb95345ac75b84263006b9b0eec8d5aa556931942589 Sep 07 11:00:30.284384 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.284217 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zqj2l" Sep 07 11:00:30.287792 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:30.287774 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36904d67_6a0b_4530_bfda_f87040d0dd71.slice/crio-61f2c2f89d8edc66f25c0a92eaffe1f80bb624ac30d35caa4f7988cf891070e1 WatchSource:0}: Error finding container 61f2c2f89d8edc66f25c0a92eaffe1f80bb624ac30d35caa4f7988cf891070e1: Status 404 returned error can't find the container with id 61f2c2f89d8edc66f25c0a92eaffe1f80bb624ac30d35caa4f7988cf891070e1 Sep 07 11:00:30.290343 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.290320 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nv7jl" Sep 07 11:00:30.297413 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:30.297390 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b81c80c_61cb_43ce_ba52_11ba84bc5a12.slice/crio-e739bbcec3e84095cf50ca0ea65d9327e05fd46e14aa019bf790e8fbb4367ec9 WatchSource:0}: Error finding container e739bbcec3e84095cf50ca0ea65d9327e05fd46e14aa019bf790e8fbb4367ec9: Status 404 returned error can't find the container with id e739bbcec3e84095cf50ca0ea65d9327e05fd46e14aa019bf790e8fbb4367ec9 Sep 07 11:00:30.301995 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:00:30.301977 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fc1277_37e8_4db4_941b_5e6c19a9a3c4.slice/crio-11c278ca96866e761340e07ff04bdb8d617944855cfe6705bf2c5606c592d4a1 WatchSource:0}: Error finding container 11c278ca96866e761340e07ff04bdb8d617944855cfe6705bf2c5606c592d4a1: Status 404 returned error can't find the container with id 11c278ca96866e761340e07ff04bdb8d617944855cfe6705bf2c5606c592d4a1 Sep 07 11:00:30.561677 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.561584 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:30.561677 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.561667 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:30.561890 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.561768 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:30.561890 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.561796 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:30.561890 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.561838 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret podName:7fbcd4d6-16cc-4ba2-933f-6938932d59a3 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:31.561818304 +0000 UTC m=+4.036174666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret") pod "global-pull-secret-syncer-4vbhj" (UID: "7fbcd4d6-16cc-4ba2-933f-6938932d59a3") : object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:30.561890 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.561859 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs podName:cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:31.56184875 +0000 UTC m=+4.036205110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs") pod "network-metrics-daemon-6jc59" (UID: "cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:30.662661 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.662624 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwrm\" (UniqueName: \"kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm\") pod \"network-check-target-qz5nc\" (UID: \"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084\") " pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:30.662853 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.662834 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 11:00:30.662936 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.662863 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 11:00:30.662936 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.662876 2571 projected.go:196] Error preparing data for projected volume kube-api-access-gzwrm for pod openshift-network-diagnostics/network-check-target-qz5nc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:30.663001 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:30.662939 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm podName:fc08ced6-7a36-4ef2-b8e2-7cd3eb377084 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:31.662920932 +0000 UTC m=+4.137277290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwrm" (UniqueName: "kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm") pod "network-check-target-qz5nc" (UID: "fc08ced6-7a36-4ef2-b8e2-7cd3eb377084") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:30.980726 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.980609 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:55:29 +0000 UTC" deadline="2028-02-11 06:15:47.437953337 +0000 UTC" Sep 07 11:00:30.980726 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:30.980686 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12523h15m16.457280526s" Sep 07 11:00:31.118404 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.118368 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:31.118592 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:31.118498 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:31.127145 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.127115 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zqj2l" event={"ID":"9b81c80c-61cb-43ce-ba52-11ba84bc5a12","Type":"ContainerStarted","Data":"e739bbcec3e84095cf50ca0ea65d9327e05fd46e14aa019bf790e8fbb4367ec9"} Sep 07 11:00:31.128455 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.128418 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p2db7" event={"ID":"36904d67-6a0b-4530-bfda-f87040d0dd71","Type":"ContainerStarted","Data":"61f2c2f89d8edc66f25c0a92eaffe1f80bb624ac30d35caa4f7988cf891070e1"} Sep 07 11:00:31.129738 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.129707 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m6xpc" event={"ID":"17610838-7a01-4354-970a-3aa13a7e64f9","Type":"ContainerStarted","Data":"d4de4b6d3043c844c3a4083ee5aafc3f292d6a75f954880e07c418bcad52d1c6"} Sep 07 11:00:31.130982 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.130963 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4fw4g" event={"ID":"c839fa80-c4fe-4444-b4d3-4de7cef50ec6","Type":"ContainerStarted","Data":"6e2ed349418eb0949df322be5b34aa1f759a42a05b988922cfae112d8261fb50"} Sep 07 11:00:31.132308 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.132285 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zh2px" event={"ID":"d4897f70-7801-4ebe-a9ff-b1df36797e92","Type":"ContainerStarted","Data":"620ca35a642afd7d97d16cb2d16bd53cd3b8e0edf823afb8674f7ecff869abcb"} Sep 07 11:00:31.133367 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.133347 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv7jl" event={"ID":"54fc1277-37e8-4db4-941b-5e6c19a9a3c4","Type":"ContainerStarted","Data":"11c278ca96866e761340e07ff04bdb8d617944855cfe6705bf2c5606c592d4a1"} Sep 07 11:00:31.134620 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.134572 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" event={"ID":"522d0108-c655-4fee-b481-82f8570276b2","Type":"ContainerStarted","Data":"58237ca26e5e966952cffb95345ac75b84263006b9b0eec8d5aa556931942589"} Sep 07 11:00:31.135882 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.135857 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" event={"ID":"484ea397-1fd4-44c3-8295-9c1833163ccc","Type":"ContainerStarted","Data":"94021bd430680b3a901ca23c787d07f784f357adb966eebeaacd1b01e7081fbc"} Sep 07 11:00:31.137032 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.137010 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r5q7t" event={"ID":"973302dc-6097-4017-b889-c7bcd790d603","Type":"ContainerStarted","Data":"adbc94f565edf89af211ecf2c3b20d3133e740936d05b80762d3452892c75e75"} Sep 07 11:00:31.569865 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.569817 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:31.570054 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.569914 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:31.570054 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:31.570041 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:31.570162 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:31.570100 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret podName:7fbcd4d6-16cc-4ba2-933f-6938932d59a3 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:33.570083548 +0000 UTC m=+6.044439924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret") pod "global-pull-secret-syncer-4vbhj" (UID: "7fbcd4d6-16cc-4ba2-933f-6938932d59a3") : object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:31.570573 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:31.570508 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:31.570729 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:31.570587 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs podName:cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:33.570568138 +0000 UTC m=+6.044924512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs") pod "network-metrics-daemon-6jc59" (UID: "cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:31.670877 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.670814 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwrm\" (UniqueName: \"kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm\") pod \"network-check-target-qz5nc\" (UID: \"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084\") " pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:31.671116 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:31.671098 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 11:00:31.671186 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:31.671126 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 11:00:31.671186 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:31.671138 2571 projected.go:196] Error preparing data for projected volume kube-api-access-gzwrm for pod openshift-network-diagnostics/network-check-target-qz5nc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:31.671263 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:31.671199 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm podName:fc08ced6-7a36-4ef2-b8e2-7cd3eb377084 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:33.671180388 +0000 UTC m=+6.145536752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwrm" (UniqueName: "kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm") pod "network-check-target-qz5nc" (UID: "fc08ced6-7a36-4ef2-b8e2-7cd3eb377084") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:31.981854 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.981714 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:55:29 +0000 UTC" deadline="2028-03-08 01:17:05.180797389 +0000 UTC" Sep 07 11:00:31.981854 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:31.981779 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13142h16m33.199021889s" Sep 07 11:00:32.121487 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:32.120725 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:32.121487 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:32.120850 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:32.121487 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:32.121277 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:32.121487 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:32.121428 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:33.119180 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:33.118879 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:33.119738 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:33.119243 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:33.158566 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:33.158509 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.ec2.internal" event={"ID":"77bb3ed25ee7e95473707df13e75eca1","Type":"ContainerStarted","Data":"c33a83a12fd5ee0af4fe922ef2a28a50623029992e98a454dd15c41cb82b9834"} Sep 07 11:00:33.172037 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:33.172000 2571 generic.go:356] "Generic (PLEG): container finished" podID="57f2b351dd062a0c56b2c5fe55d5b879" containerID="a513aa5a04cc79790caae8049ac2c1cf463adefdfa62c0ee52291161dc4ffba1" exitCode=0 Sep 07 11:00:33.172223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:33.172059 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" event={"ID":"57f2b351dd062a0c56b2c5fe55d5b879","Type":"ContainerDied","Data":"a513aa5a04cc79790caae8049ac2c1cf463adefdfa62c0ee52291161dc4ffba1"} Sep 07 11:00:33.190442 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:33.190354 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.ec2.internal" podStartSLOduration=4.190332481 podStartE2EDuration="4.190332481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:33.174034266 +0000 UTC m=+5.648390650" watchObservedRunningTime="2026-09-07 11:00:33.190332481 +0000 UTC m=+5.664688861" Sep 07 11:00:33.591631 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:33.590833 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:33.591631 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:33.590902 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:33.591631 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:33.591040 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:33.591631 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:33.591104 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs podName:cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:37.591084363 +0000 UTC m=+10.065440719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs") pod "network-metrics-daemon-6jc59" (UID: "cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:33.591631 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:33.591483 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:33.591631 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:33.591534 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret podName:7fbcd4d6-16cc-4ba2-933f-6938932d59a3 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:37.591518773 +0000 UTC m=+10.065875146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret") pod "global-pull-secret-syncer-4vbhj" (UID: "7fbcd4d6-16cc-4ba2-933f-6938932d59a3") : object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:33.692020 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:33.691344 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwrm\" (UniqueName: \"kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm\") pod \"network-check-target-qz5nc\" (UID: \"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084\") " pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:33.692020 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:33.691568 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 11:00:33.692020 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:33.691590 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 11:00:33.692020 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:33.691603 2571 projected.go:196] Error preparing data for projected volume kube-api-access-gzwrm for pod openshift-network-diagnostics/network-check-target-qz5nc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:33.692020 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:33.691664 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm podName:fc08ced6-7a36-4ef2-b8e2-7cd3eb377084 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:37.691645978 +0000 UTC m=+10.166002335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwrm" (UniqueName: "kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm") pod "network-check-target-qz5nc" (UID: "fc08ced6-7a36-4ef2-b8e2-7cd3eb377084") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:34.119125 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:34.118082 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:34.119125 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:34.118217 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:34.119125 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:34.118722 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:34.119125 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:34.118817 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:35.118802 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:35.118770 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:35.118967 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:35.118892 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:36.118937 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:36.118880 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:36.119422 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:36.119008 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:36.119422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:36.119074 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:36.119422 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:36.119189 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:37.118444 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:37.118409 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:37.118643 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:37.118570 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:37.627313 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:37.626533 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:37.627313 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:37.626645 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:37.627313 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:37.626776 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:37.627313 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:37.626831 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs podName:cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:45.626818336 +0000 UTC m=+18.101174705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs") pod "network-metrics-daemon-6jc59" (UID: "cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:37.627313 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:37.627212 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:37.627313 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:37.627270 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret podName:7fbcd4d6-16cc-4ba2-933f-6938932d59a3 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:45.62725542 +0000 UTC m=+18.101611783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret") pod "global-pull-secret-syncer-4vbhj" (UID: "7fbcd4d6-16cc-4ba2-933f-6938932d59a3") : object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:37.727421 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:37.727382 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwrm\" (UniqueName: \"kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm\") pod \"network-check-target-qz5nc\" (UID: \"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084\") " pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:37.727627 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:37.727523 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 11:00:37.727627 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:37.727568 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 11:00:37.727627 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:37.727584 2571 projected.go:196] Error preparing data for projected volume kube-api-access-gzwrm for pod openshift-network-diagnostics/network-check-target-qz5nc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:37.727791 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:37.727646 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm podName:fc08ced6-7a36-4ef2-b8e2-7cd3eb377084 nodeName:}" failed. No retries permitted until 2026-09-07 11:00:45.727628073 +0000 UTC m=+18.201984467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwrm" (UniqueName: "kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm") pod "network-check-target-qz5nc" (UID: "fc08ced6-7a36-4ef2-b8e2-7cd3eb377084") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:38.119116 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:38.119039 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:38.119275 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:38.119155 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:38.119473 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:38.119446 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:38.119625 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:38.119602 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:39.118867 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:39.118836 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:39.119335 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:39.118947 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:40.121930 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:40.121900 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:40.122353 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:40.122035 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:40.122353 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:40.122087 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:40.122353 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:40.122190 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:41.118175 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:41.118089 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:41.118324 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:41.118219 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:42.118495 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:42.118458 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:42.118495 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:42.118477 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:42.119022 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:42.118607 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:42.119022 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:42.118738 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:43.118755 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:43.118720 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:43.119186 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:43.118851 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:44.121296 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:44.121267 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:44.121703 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:44.121371 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:44.121703 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:44.121417 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:44.121703 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:44.121497 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:45.118536 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:45.118499 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:45.118774 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:45.118658 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:45.686507 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:45.686467 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:45.686908 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:45.686567 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:45.686908 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:45.686632 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:45.686908 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:45.686666 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:45.686908 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:45.686706 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs podName:cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:01.686687003 +0000 UTC m=+34.161043373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs") pod "network-metrics-daemon-6jc59" (UID: "cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:00:45.686908 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:45.686724 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret podName:7fbcd4d6-16cc-4ba2-933f-6938932d59a3 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:01.686715779 +0000 UTC m=+34.161072135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret") pod "global-pull-secret-syncer-4vbhj" (UID: "7fbcd4d6-16cc-4ba2-933f-6938932d59a3") : object "kube-system"/"original-pull-secret" not registered Sep 07 11:00:45.787377 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:45.787317 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwrm\" (UniqueName: \"kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm\") pod \"network-check-target-qz5nc\" (UID: \"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084\") " pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:45.787593 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:45.787519 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 11:00:45.787593 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:45.787564 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 11:00:45.787593 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:45.787577 2571 projected.go:196] Error preparing data for projected volume kube-api-access-gzwrm for pod openshift-network-diagnostics/network-check-target-qz5nc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:45.787758 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:45.787644 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm podName:fc08ced6-7a36-4ef2-b8e2-7cd3eb377084 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:01.787630385 +0000 UTC m=+34.261986766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwrm" (UniqueName: "kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm") pod "network-check-target-qz5nc" (UID: "fc08ced6-7a36-4ef2-b8e2-7cd3eb377084") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:00:46.118243 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:46.118167 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:46.118243 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:46.118190 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:46.118457 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:46.118302 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:46.118457 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:46.118429 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:47.118512 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:47.118477 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:47.118992 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:47.118628 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:48.119204 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:48.119178 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:48.119543 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:48.119275 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:48.119543 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:48.119330 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:48.119543 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:48.119379 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:49.118656 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.118317 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:49.118803 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:49.118768 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:49.207413 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.207341 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" event={"ID":"522d0108-c655-4fee-b481-82f8570276b2","Type":"ContainerStarted","Data":"2752533006e539501897a602116128cacf70ab3809a3e5e840e3c7020fbe83ca"} Sep 07 11:00:49.210416 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.210387 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" event={"ID":"484ea397-1fd4-44c3-8295-9c1833163ccc","Type":"ContainerStarted","Data":"78fcc667ef818b51256e779d896d4204e8eca2f341f96e8ec51e4d15d3482046"} Sep 07 11:00:49.210544 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.210474 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" event={"ID":"484ea397-1fd4-44c3-8295-9c1833163ccc","Type":"ContainerStarted","Data":"8e7d6407856fec87f3c231ce64d60f4ef34ff8ba57619eb354db828b120ab7c9"} Sep 07 11:00:49.210544 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.210525 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" event={"ID":"484ea397-1fd4-44c3-8295-9c1833163ccc","Type":"ContainerStarted","Data":"f1ed20b7fd83b78b017586d778ad9eb12b44fd884256f3f0611a1627b2910fdb"} Sep 07 11:00:49.210661 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.210587 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" event={"ID":"484ea397-1fd4-44c3-8295-9c1833163ccc","Type":"ContainerStarted","Data":"db946c4191779d07eff6e38218c3f33946f02a06f501a12fe7f96a4f0e700350"} Sep 07 11:00:49.210661 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.210634 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" event={"ID":"484ea397-1fd4-44c3-8295-9c1833163ccc","Type":"ContainerStarted","Data":"b2d279f3db4fba281d65f9f3e031f939774e210664027ceadb98a0d631d85e7f"} Sep 07 11:00:49.210741 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.210678 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" event={"ID":"484ea397-1fd4-44c3-8295-9c1833163ccc","Type":"ContainerStarted","Data":"071d6daa4796cd92d97d2bf2fc4bcc01c0fa97b2c5cf6aa7890dfc5ffba0e464"} Sep 07 11:00:49.211847 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.211814 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r5q7t" event={"ID":"973302dc-6097-4017-b889-c7bcd790d603","Type":"ContainerStarted","Data":"3a04242853844aee47dd291839a048254248dfe74c2e3fce29d7d503dc635c81"} Sep 07 11:00:49.213254 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.213234 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p2db7" event={"ID":"36904d67-6a0b-4530-bfda-f87040d0dd71","Type":"ContainerStarted","Data":"4a05e29c0f03b46d992fd9e30b2d10a646daabd032c13c78d2dc5ec0a0269e49"} Sep 07 11:00:49.214687 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.214663 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m6xpc" event={"ID":"17610838-7a01-4354-970a-3aa13a7e64f9","Type":"ContainerStarted","Data":"0a4691ea382a387f714a2a51df060e1c41a73b641f47a4ecafec4ae240e962b1"} Sep 07 11:00:49.216124 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.216099 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4fw4g" event={"ID":"c839fa80-c4fe-4444-b4d3-4de7cef50ec6","Type":"ContainerStarted","Data":"0dc4c63b1f1820370b41ba0ed58bc5fd01f31f1738f6642a5025fd41cfa689e6"} Sep 07 11:00:49.217931 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.217908 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zh2px" event={"ID":"d4897f70-7801-4ebe-a9ff-b1df36797e92","Type":"ContainerStarted","Data":"bbcba7f3faed816f692f04dabb1e02b172417cd156b1955eb6742f5810735a28"} Sep 07 11:00:49.219678 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.219649 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" event={"ID":"57f2b351dd062a0c56b2c5fe55d5b879","Type":"ContainerStarted","Data":"e80f278b65ac3f22ea549decdebe0a37367c240af63ec73446c5c72d72a27646"} Sep 07 11:00:49.221073 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.221055 2571 generic.go:356] "Generic (PLEG): container finished" podID="54fc1277-37e8-4db4-941b-5e6c19a9a3c4" containerID="546f6afb0e138a4d22888a65ffe86eef7ddddffa2abc099e0f20fbdb934f1674" exitCode=0 Sep 07 11:00:49.221147 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.221081 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv7jl" event={"ID":"54fc1277-37e8-4db4-941b-5e6c19a9a3c4","Type":"ContainerDied","Data":"546f6afb0e138a4d22888a65ffe86eef7ddddffa2abc099e0f20fbdb934f1674"} Sep 07 11:00:49.227441 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.227404 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r5q7t" podStartSLOduration=3.236386976 podStartE2EDuration="21.227393368s" totalImagesPullingTime="17.991006392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:49.227031239 +0000 UTC m=+21.701387620" watchObservedRunningTime="2026-09-07 11:00:49.227393368 +0000 UTC m=+21.701749745" Sep 07 11:00:49.242319 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.242266 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zh2px" podStartSLOduration=3.231301764 podStartE2EDuration="21.242247871s" totalImagesPullingTime="18.010946107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:49.241942815 +0000 UTC m=+21.716299186" watchObservedRunningTime="2026-09-07 11:00:49.242247871 +0000 UTC m=+21.716604249" Sep 07 11:00:49.274100 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.274054 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4fw4g" podStartSLOduration=3.270510141 podStartE2EDuration="21.274033986s" totalImagesPullingTime="18.003523845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:49.257351522 +0000 UTC m=+21.731707898" watchObservedRunningTime="2026-09-07 11:00:49.274033986 +0000 UTC m=+21.748390363" Sep 07 11:00:49.274260 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.274175 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m6xpc" podStartSLOduration=3.096878146 podStartE2EDuration="21.274170181s" totalImagesPullingTime="18.177292035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:49.274032878 +0000 UTC m=+21.748389256" watchObservedRunningTime="2026-09-07 11:00:49.274170181 +0000 UTC m=+21.748526559" Sep 07 11:00:49.288110 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.288056 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p2db7" podStartSLOduration=10.723382997 podStartE2EDuration="21.288034688s" totalImagesPullingTime="10.564651691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:49.287929997 +0000 UTC m=+21.762286375" watchObservedRunningTime="2026-09-07 11:00:49.288034688 +0000 UTC m=+21.762391066" Sep 07 11:00:49.324812 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.324752 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.ec2.internal" podStartSLOduration=20.324732549 podStartE2EDuration="20.324732549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:49.324617077 +0000 UTC m=+21.798973456" watchObservedRunningTime="2026-09-07 11:00:49.324732549 +0000 UTC m=+21.799088926" Sep 07 11:00:49.494669 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:49.494645 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 07 11:00:50.011599 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:50.011419 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T11:00:49.494666047Z","UUID":"2cf40c2a-bd2a-4c18-81d6-5d5a17b67326","Handler":null,"Name":"","Endpoint":""} Sep 07 11:00:50.014621 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:50.014596 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 07 11:00:50.014754 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:50.014636 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 07 11:00:50.118493 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:50.118458 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:50.118683 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:50.118458 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:50.118683 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:50.118587 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:50.118683 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:50.118656 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:50.226044 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:50.226004 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" event={"ID":"522d0108-c655-4fee-b481-82f8570276b2","Type":"ContainerStarted","Data":"23ecdc3c211cff345f9182c066db2fa74674151188d0f090e173ba0a81d583f5"} Sep 07 11:00:50.227966 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:50.227933 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zqj2l" event={"ID":"9b81c80c-61cb-43ce-ba52-11ba84bc5a12","Type":"ContainerStarted","Data":"214fcdb5200926af10dcca41e6260bcb36702ff65545469b320e6f3a6480c033"} Sep 07 11:00:50.245955 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:50.245884 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zqj2l" podStartSLOduration=4.300770208 podStartE2EDuration="22.245863858s" totalImagesPullingTime="17.94509365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:50.245663411 +0000 UTC m=+22.720019793" watchObservedRunningTime="2026-09-07 11:00:50.245863858 +0000 UTC m=+22.720220240" Sep 07 11:00:51.118159 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:51.118125 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:51.118320 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:51.118278 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:51.232583 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:51.232481 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" event={"ID":"522d0108-c655-4fee-b481-82f8570276b2","Type":"ContainerStarted","Data":"7147cbc6a2dd43ca28ffacad6849a06a512f6f124938fbdc3f2d524fbdf234fd"} Sep 07 11:00:51.236333 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:51.236302 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" event={"ID":"484ea397-1fd4-44c3-8295-9c1833163ccc","Type":"ContainerStarted","Data":"c84fd41428f9f9641b58342da0e2e4c47d3f00c4a989defc475bb256d1aac772"} Sep 07 11:00:51.254901 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:51.254833 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qs77" podStartSLOduration=3.384243453 podStartE2EDuration="23.254817318s" totalImagesPullingTime="19.870573865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:51.254747123 +0000 UTC m=+23.729103507" watchObservedRunningTime="2026-09-07 11:00:51.254817318 +0000 UTC m=+23.729173692" Sep 07 11:00:51.932566 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:51.932516 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:51.933151 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:51.933125 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:52.118812 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:52.118779 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:52.118812 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:52.118779 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:52.119012 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:52.118884 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:52.119048 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:52.119011 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:52.238825 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:52.238583 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:52.239150 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:52.238930 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p2db7" Sep 07 11:00:53.118436 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:53.118404 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:53.118605 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:53.118517 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:54.118324 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:54.118282 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:54.118324 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:54.118328 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:54.118798 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:54.118415 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:54.118798 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:54.118511 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:55.118785 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:55.118756 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:55.119473 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:55.118864 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:55.245399 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:55.245369 2571 generic.go:356] "Generic (PLEG): container finished" podID="54fc1277-37e8-4db4-941b-5e6c19a9a3c4" containerID="699cb1ef9408e1fdec475183e5aff536c97749710e229209fc722dc411bcb084" exitCode=0 Sep 07 11:00:55.245592 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:55.245442 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv7jl" event={"ID":"54fc1277-37e8-4db4-941b-5e6c19a9a3c4","Type":"ContainerDied","Data":"699cb1ef9408e1fdec475183e5aff536c97749710e229209fc722dc411bcb084"} Sep 07 11:00:55.249069 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:55.249046 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" event={"ID":"484ea397-1fd4-44c3-8295-9c1833163ccc","Type":"ContainerStarted","Data":"4c7cd3cea76f61e781006cb6d235a498179c26deda3932a16e22b721c4c75977"} Sep 07 11:00:55.249333 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:55.249315 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:55.249430 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:55.249396 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:55.264525 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:55.264500 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:55.289778 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:55.289729 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" podStartSLOduration=9.284613152 podStartE2EDuration="27.289713431s" totalImagesPullingTime="18.005100279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:55.28934526 +0000 UTC m=+27.763701640" watchObservedRunningTime="2026-09-07 11:00:55.289713431 +0000 UTC m=+27.764069809" Sep 07 11:00:56.119215 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:56.119024 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:56.119685 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:56.119070 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:56.119685 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:56.119316 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:56.119685 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:56.119387 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:56.171392 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:56.171365 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4vbhj"] Sep 07 11:00:56.174033 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:56.173996 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qz5nc"] Sep 07 11:00:56.185368 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:56.185335 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jc59"] Sep 07 11:00:56.185480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:56.185449 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:56.185605 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:56.185583 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:56.253716 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:56.253684 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:56.253867 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:56.253792 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:56.253867 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:56.253793 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:56.254169 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:56.254150 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:56.254234 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:56.254184 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:56.268784 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:56.268758 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:00:57.257908 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:57.257874 2571 generic.go:356] "Generic (PLEG): container finished" podID="54fc1277-37e8-4db4-941b-5e6c19a9a3c4" containerID="93a128c2e62caf76d3c0ac2ddaaced11d2c3a3b10a764eb9a4b84a52d3f16cee" exitCode=0 Sep 07 11:00:57.258246 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:57.257962 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv7jl" event={"ID":"54fc1277-37e8-4db4-941b-5e6c19a9a3c4","Type":"ContainerDied","Data":"93a128c2e62caf76d3c0ac2ddaaced11d2c3a3b10a764eb9a4b84a52d3f16cee"} Sep 07 11:00:58.119490 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:58.119413 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:00:58.119490 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:58.119452 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:00:58.119700 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:58.119523 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:00:58.119700 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:58.119599 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:00:58.119700 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:58.119652 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:00:58.119791 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:00:58.119725 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:00:58.262488 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:58.262455 2571 generic.go:356] "Generic (PLEG): container finished" podID="54fc1277-37e8-4db4-941b-5e6c19a9a3c4" containerID="15f9dbdef41a8b188de0744204a3b51271ff94e8e81b43aa7e10c673ea899d7d" exitCode=0 Sep 07 11:00:58.262837 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:00:58.262534 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv7jl" event={"ID":"54fc1277-37e8-4db4-941b-5e6c19a9a3c4","Type":"ContainerDied","Data":"15f9dbdef41a8b188de0744204a3b51271ff94e8e81b43aa7e10c673ea899d7d"} Sep 07 11:01:00.118938 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:00.118653 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:01:00.119377 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:00.118653 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:01:00.119377 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:00.118716 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:01:00.119377 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:00.119056 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:01:00.119377 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:00.119177 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:01:00.119377 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:00.119258 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:01:01.709084 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:01.709042 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:01:01.709611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:01.709115 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:01:01.709611 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:01.709194 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 11:01:01.709611 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:01.709215 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:01:01.709611 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:01.709278 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret podName:7fbcd4d6-16cc-4ba2-933f-6938932d59a3 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:33.709255424 +0000 UTC m=+66.183611808 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret") pod "global-pull-secret-syncer-4vbhj" (UID: "7fbcd4d6-16cc-4ba2-933f-6938932d59a3") : object "kube-system"/"original-pull-secret" not registered Sep 07 11:01:01.709611 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:01.709299 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs podName:cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:33.709290511 +0000 UTC m=+66.183646867 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs") pod "network-metrics-daemon-6jc59" (UID: "cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 11:01:01.810417 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:01.810380 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwrm\" (UniqueName: \"kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm\") pod \"network-check-target-qz5nc\" (UID: \"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084\") " pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:01:01.810630 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:01.810574 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 11:01:01.810630 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:01.810614 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 11:01:01.810630 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:01.810629 2571 projected.go:196] Error preparing data for projected volume kube-api-access-gzwrm for pod openshift-network-diagnostics/network-check-target-qz5nc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:01:01.810819 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:01.810689 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm podName:fc08ced6-7a36-4ef2-b8e2-7cd3eb377084 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:33.810674991 +0000 UTC m=+66.285031347 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzwrm" (UniqueName: "kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm") pod "network-check-target-qz5nc" (UID: "fc08ced6-7a36-4ef2-b8e2-7cd3eb377084") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 11:01:02.118535 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.118467 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:01:02.118743 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.118467 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:01:02.118743 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:02.118632 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jc59" podUID="cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0" Sep 07 11:01:02.118743 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:02.118677 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz5nc" podUID="fc08ced6-7a36-4ef2-b8e2-7cd3eb377084" Sep 07 11:01:02.118743 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.118481 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:01:02.118954 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:02.118805 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vbhj" podUID="7fbcd4d6-16cc-4ba2-933f-6938932d59a3" Sep 07 11:01:02.376942 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.376855 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-34.ec2.internal" event="NodeReady" Sep 07 11:01:02.377097 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.377009 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 07 11:01:02.409451 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.409416 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dd4d7ff94-btjx6"] Sep 07 11:01:02.441023 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.440992 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-49lhf"] Sep 07 11:01:02.441194 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.441130 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.444038 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.443602 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 07 11:01:02.444038 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.443719 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rmbnm\"" Sep 07 11:01:02.444038 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.443775 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 07 11:01:02.444038 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.443896 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 07 11:01:02.450656 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.450637 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 07 11:01:02.455997 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.455964 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dd4d7ff94-btjx6"] Sep 07 11:01:02.456134 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.456004 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sjcrj"] Sep 07 11:01:02.456427 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.456403 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.458841 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.458812 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 07 11:01:02.458955 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.458913 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-czp2d\"" Sep 07 11:01:02.459016 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.458913 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 07 11:01:02.477665 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.477641 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-49lhf"] Sep 07 11:01:02.477665 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.477674 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjcrj"] Sep 07 11:01:02.477836 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.477751 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjcrj" Sep 07 11:01:02.480505 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.480480 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 07 11:01:02.480641 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.480607 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 07 11:01:02.480727 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.480710 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2qqrh\"" Sep 07 11:01:02.480924 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.480908 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 07 11:01:02.514934 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.514902 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-installation-pull-secrets\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.515087 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.514939 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-bound-sa-token\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.515087 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.514965 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrd4x\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-kube-api-access-rrd4x\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.515087 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.515038 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-image-registry-private-configuration\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.515203 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.515100 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a201c33-d8a9-4792-9add-57056228abf6-ca-trust-extracted\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.515203 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.515122 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.515203 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.515138 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-trusted-ca\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.515203 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.515155 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-registry-certificates\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.616272 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616234 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-installation-pull-secrets\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.616272 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616276 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-bound-sa-token\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.616491 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616294 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrd4x\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-kube-api-access-rrd4x\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.616491 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616332 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-image-registry-private-configuration\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.616491 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616360 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd09d09c-a0d7-4104-8d79-746a7d0b8030-config-volume\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.616491 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616400 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a201c33-d8a9-4792-9add-57056228abf6-ca-trust-extracted\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.616491 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616427 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.616491 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616451 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-trusted-ca\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.616491 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616472 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62n9s\" (UniqueName: \"kubernetes.io/projected/e23c9000-ee54-41ff-8409-5da16eed5f60-kube-api-access-62n9s\") pod \"ingress-canary-sjcrj\" (UID: \"e23c9000-ee54-41ff-8409-5da16eed5f60\") " pod="openshift-ingress-canary/ingress-canary-sjcrj" Sep 07 11:01:02.616857 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-registry-certificates\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.616857 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616518 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e23c9000-ee54-41ff-8409-5da16eed5f60-cert\") pod \"ingress-canary-sjcrj\" (UID: \"e23c9000-ee54-41ff-8409-5da16eed5f60\") " pod="openshift-ingress-canary/ingress-canary-sjcrj" Sep 07 11:01:02.616857 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:02.616539 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 11:01:02.616857 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:02.616582 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd4d7ff94-btjx6: secret "image-registry-tls" not found Sep 07 11:01:02.616857 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.616857 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:02.616636 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls podName:5a201c33-d8a9-4792-9add-57056228abf6 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:03.116618184 +0000 UTC m=+35.590974540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls") pod "image-registry-dd4d7ff94-btjx6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6") : secret "image-registry-tls" not found Sep 07 11:01:02.616857 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616711 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd09d09c-a0d7-4104-8d79-746a7d0b8030-tmp-dir\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.616857 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616745 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmfmd\" (UniqueName: \"kubernetes.io/projected/cd09d09c-a0d7-4104-8d79-746a7d0b8030-kube-api-access-nmfmd\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.617149 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.616980 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a201c33-d8a9-4792-9add-57056228abf6-ca-trust-extracted\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.617192 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.617167 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-registry-certificates\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.617416 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.617395 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-trusted-ca\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.621217 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.621198 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-installation-pull-secrets\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.621317 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.621200 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-image-registry-private-configuration\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.624766 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.624737 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-bound-sa-token\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.624951 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.624924 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrd4x\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-kube-api-access-rrd4x\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:02.717686 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.717589 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62n9s\" (UniqueName: \"kubernetes.io/projected/e23c9000-ee54-41ff-8409-5da16eed5f60-kube-api-access-62n9s\") pod \"ingress-canary-sjcrj\" (UID: \"e23c9000-ee54-41ff-8409-5da16eed5f60\") " pod="openshift-ingress-canary/ingress-canary-sjcrj" Sep 07 11:01:02.717686 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.717631 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e23c9000-ee54-41ff-8409-5da16eed5f60-cert\") pod \"ingress-canary-sjcrj\" (UID: \"e23c9000-ee54-41ff-8409-5da16eed5f60\") " pod="openshift-ingress-canary/ingress-canary-sjcrj" Sep 07 11:01:02.717686 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.717656 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.717686 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.717681 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd09d09c-a0d7-4104-8d79-746a7d0b8030-tmp-dir\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.718442 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.717706 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmfmd\" (UniqueName: \"kubernetes.io/projected/cd09d09c-a0d7-4104-8d79-746a7d0b8030-kube-api-access-nmfmd\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.718442 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:02.717756 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 11:01:02.718442 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.717778 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd09d09c-a0d7-4104-8d79-746a7d0b8030-config-volume\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.718442 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:02.717820 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls podName:cd09d09c-a0d7-4104-8d79-746a7d0b8030 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:03.2178007 +0000 UTC m=+35.692157060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls") pod "dns-default-49lhf" (UID: "cd09d09c-a0d7-4104-8d79-746a7d0b8030") : secret "dns-default-metrics-tls" not found Sep 07 11:01:02.718442 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.718011 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd09d09c-a0d7-4104-8d79-746a7d0b8030-tmp-dir\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.718442 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.718285 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd09d09c-a0d7-4104-8d79-746a7d0b8030-config-volume\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.719962 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.719939 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e23c9000-ee54-41ff-8409-5da16eed5f60-cert\") pod \"ingress-canary-sjcrj\" (UID: \"e23c9000-ee54-41ff-8409-5da16eed5f60\") " pod="openshift-ingress-canary/ingress-canary-sjcrj" Sep 07 11:01:02.726184 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.726158 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62n9s\" (UniqueName: \"kubernetes.io/projected/e23c9000-ee54-41ff-8409-5da16eed5f60-kube-api-access-62n9s\") pod \"ingress-canary-sjcrj\" (UID: \"e23c9000-ee54-41ff-8409-5da16eed5f60\") " pod="openshift-ingress-canary/ingress-canary-sjcrj" Sep 07 11:01:02.726422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.726394 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmfmd\" (UniqueName: \"kubernetes.io/projected/cd09d09c-a0d7-4104-8d79-746a7d0b8030-kube-api-access-nmfmd\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:02.785689 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:02.785647 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjcrj" Sep 07 11:01:03.124251 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:03.124211 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:03.124430 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:03.124407 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 11:01:03.124494 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:03.124434 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd4d7ff94-btjx6: secret "image-registry-tls" not found Sep 07 11:01:03.124543 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:03.124516 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls podName:5a201c33-d8a9-4792-9add-57056228abf6 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:04.124495214 +0000 UTC m=+36.598851593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls") pod "image-registry-dd4d7ff94-btjx6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6") : secret "image-registry-tls" not found Sep 07 11:01:03.225569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:03.225454 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:03.225702 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:03.225650 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 11:01:03.225756 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:03.225729 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls podName:cd09d09c-a0d7-4104-8d79-746a7d0b8030 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:04.225707914 +0000 UTC m=+36.700064290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls") pod "dns-default-49lhf" (UID: "cd09d09c-a0d7-4104-8d79-746a7d0b8030") : secret "dns-default-metrics-tls" not found Sep 07 11:01:03.239710 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:03.239685 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjcrj"] Sep 07 11:01:03.246462 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:01:03.246430 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23c9000_ee54_41ff_8409_5da16eed5f60.slice/crio-d380540aed309dd3ee198c31fa00c0e3c28aed26965d9ca82d53484ac0fe67a6 WatchSource:0}: Error finding container d380540aed309dd3ee198c31fa00c0e3c28aed26965d9ca82d53484ac0fe67a6: Status 404 returned error can't find the container with id d380540aed309dd3ee198c31fa00c0e3c28aed26965d9ca82d53484ac0fe67a6 Sep 07 11:01:03.276025 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:03.275997 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjcrj" event={"ID":"e23c9000-ee54-41ff-8409-5da16eed5f60","Type":"ContainerStarted","Data":"d380540aed309dd3ee198c31fa00c0e3c28aed26965d9ca82d53484ac0fe67a6"} Sep 07 11:01:03.278723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:03.278695 2571 generic.go:356] "Generic (PLEG): container finished" podID="54fc1277-37e8-4db4-941b-5e6c19a9a3c4" containerID="9eb93e03a474cfb5318e82a9f736bac4cccc01301d3f4ef94520c4dcfa08a572" exitCode=0 Sep 07 11:01:03.278824 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:03.278728 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv7jl" event={"ID":"54fc1277-37e8-4db4-941b-5e6c19a9a3c4","Type":"ContainerDied","Data":"9eb93e03a474cfb5318e82a9f736bac4cccc01301d3f4ef94520c4dcfa08a572"} Sep 07 11:01:04.119032 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.118995 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:01:04.119609 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.119005 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:01:04.119609 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.119110 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:01:04.122055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.121671 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 11:01:04.122055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.121681 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 07 11:01:04.122055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.121782 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4v4fp\"" Sep 07 11:01:04.123141 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.123004 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 11:01:04.123141 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.123013 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m22ww\"" Sep 07 11:01:04.123141 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.123038 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 07 11:01:04.132862 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.132834 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:04.133239 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:04.132992 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 11:01:04.133239 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:04.133010 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd4d7ff94-btjx6: secret "image-registry-tls" not found Sep 07 11:01:04.133239 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:04.133074 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls podName:5a201c33-d8a9-4792-9add-57056228abf6 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:06.133055427 +0000 UTC m=+38.607411798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls") pod "image-registry-dd4d7ff94-btjx6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6") : secret "image-registry-tls" not found Sep 07 11:01:04.233716 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.233683 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:04.233889 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:04.233833 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 11:01:04.233952 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:04.233899 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls podName:cd09d09c-a0d7-4104-8d79-746a7d0b8030 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:06.233882005 +0000 UTC m=+38.708238382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls") pod "dns-default-49lhf" (UID: "cd09d09c-a0d7-4104-8d79-746a7d0b8030") : secret "dns-default-metrics-tls" not found Sep 07 11:01:04.284660 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.284626 2571 generic.go:356] "Generic (PLEG): container finished" podID="54fc1277-37e8-4db4-941b-5e6c19a9a3c4" containerID="97587a347517cd2865d6a01f77c6666fa616b4738074a1fce8e9864f33743707" exitCode=0 Sep 07 11:01:04.284845 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:04.284684 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv7jl" event={"ID":"54fc1277-37e8-4db4-941b-5e6c19a9a3c4","Type":"ContainerDied","Data":"97587a347517cd2865d6a01f77c6666fa616b4738074a1fce8e9864f33743707"} Sep 07 11:01:05.290912 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:05.290878 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nv7jl" event={"ID":"54fc1277-37e8-4db4-941b-5e6c19a9a3c4","Type":"ContainerStarted","Data":"e02479c4a4895bbac1d9c9c9b74abd2be9cd7afafd9ba01e74fbb05a5bcefa8f"} Sep 07 11:01:05.312315 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:05.312065 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nv7jl" podStartSLOduration=6.990030073 podStartE2EDuration="37.312043537s" totalImagesPullingTime="30.322013464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:01:05.310381457 +0000 UTC m=+37.784737855" watchObservedRunningTime="2026-09-07 11:01:05.312043537 +0000 UTC m=+37.786399916" Sep 07 11:01:06.149124 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:06.148959 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:06.149254 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:06.149160 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 11:01:06.149254 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:06.149186 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd4d7ff94-btjx6: secret "image-registry-tls" not found Sep 07 11:01:06.149336 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:06.149256 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls podName:5a201c33-d8a9-4792-9add-57056228abf6 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:10.149235538 +0000 UTC m=+42.623591912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls") pod "image-registry-dd4d7ff94-btjx6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6") : secret "image-registry-tls" not found Sep 07 11:01:06.250033 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:06.249938 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:06.250176 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:06.250081 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 11:01:06.250176 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:06.250144 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls podName:cd09d09c-a0d7-4104-8d79-746a7d0b8030 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:10.250130181 +0000 UTC m=+42.724486538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls") pod "dns-default-49lhf" (UID: "cd09d09c-a0d7-4104-8d79-746a7d0b8030") : secret "dns-default-metrics-tls" not found Sep 07 11:01:06.294749 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:06.294708 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjcrj" event={"ID":"e23c9000-ee54-41ff-8409-5da16eed5f60","Type":"ContainerStarted","Data":"4561ed3d04f6451cc1e97bf845902826de590f9388ec8f4ad1936267922ad866"} Sep 07 11:01:06.310212 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:06.310167 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sjcrj" podStartSLOduration=1.576278017 podStartE2EDuration="4.310154077s" totalImagesPullingTime="2.73387606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:01:06.310068705 +0000 UTC m=+38.784425083" watchObservedRunningTime="2026-09-07 11:01:06.310154077 +0000 UTC m=+38.784510483" Sep 07 11:01:10.174830 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:10.174792 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:10.175236 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:10.174945 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 11:01:10.175236 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:10.174960 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd4d7ff94-btjx6: secret "image-registry-tls" not found Sep 07 11:01:10.175236 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:10.175012 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls podName:5a201c33-d8a9-4792-9add-57056228abf6 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:18.174995841 +0000 UTC m=+50.649352217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls") pod "image-registry-dd4d7ff94-btjx6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6") : secret "image-registry-tls" not found Sep 07 11:01:10.275863 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:10.275826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:10.276032 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:10.275959 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 11:01:10.276032 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:10.276010 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls podName:cd09d09c-a0d7-4104-8d79-746a7d0b8030 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:18.275998356 +0000 UTC m=+50.750354712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls") pod "dns-default-49lhf" (UID: "cd09d09c-a0d7-4104-8d79-746a7d0b8030") : secret "dns-default-metrics-tls" not found Sep 07 11:01:18.237058 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:18.237010 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:18.237586 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:18.237162 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 11:01:18.237586 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:18.237181 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd4d7ff94-btjx6: secret "image-registry-tls" not found Sep 07 11:01:18.237586 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:18.237252 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls podName:5a201c33-d8a9-4792-9add-57056228abf6 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:34.237235857 +0000 UTC m=+66.711592214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls") pod "image-registry-dd4d7ff94-btjx6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6") : secret "image-registry-tls" not found Sep 07 11:01:18.337849 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:18.337812 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:18.338000 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:18.337925 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 11:01:18.338000 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:18.337981 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls podName:cd09d09c-a0d7-4104-8d79-746a7d0b8030 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:34.337968903 +0000 UTC m=+66.812325259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls") pod "dns-default-49lhf" (UID: "cd09d09c-a0d7-4104-8d79-746a7d0b8030") : secret "dns-default-metrics-tls" not found Sep 07 11:01:19.280294 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.280251 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc"] Sep 07 11:01:19.289561 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.289525 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j"] Sep 07 11:01:19.289694 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.289632 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" Sep 07 11:01:19.292336 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.292314 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 07 11:01:19.292506 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.292316 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 07 11:01:19.292745 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.292724 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 07 11:01:19.292824 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.292814 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 07 11:01:19.293071 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.293050 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pdvvx\"" Sep 07 11:01:19.299629 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.299608 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc"] Sep 07 11:01:19.299730 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.299713 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:19.302238 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.302214 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 07 11:01:19.302336 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.302266 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 07 11:01:19.302336 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.302278 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 07 11:01:19.302336 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.302306 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nqnjj\"" Sep 07 11:01:19.308646 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.308625 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j"] Sep 07 11:01:19.378522 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.378488 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2nrt8"] Sep 07 11:01:19.393284 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.393257 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2nrt8"] Sep 07 11:01:19.393433 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.393375 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2nrt8" Sep 07 11:01:19.395775 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.395751 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh"] Sep 07 11:01:19.396059 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.396036 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 07 11:01:19.396180 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.396100 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 07 11:01:19.396180 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.396100 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2ckjh\"" Sep 07 11:01:19.408958 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.408895 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh"] Sep 07 11:01:19.409081 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.408997 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" Sep 07 11:01:19.411569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.411529 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 07 11:01:19.411770 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.411525 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 07 11:01:19.411770 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.411640 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 07 11:01:19.411770 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.411727 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 07 11:01:19.411959 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.411942 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zjzmg\"" Sep 07 11:01:19.444961 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.444932 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d-serving-cert\") pod \"service-ca-operator-859f94c4b-lm4tc\" (UID: \"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" Sep 07 11:01:19.445138 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.444971 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgj2z\" (UniqueName: \"kubernetes.io/projected/ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d-kube-api-access-xgj2z\") pod \"service-ca-operator-859f94c4b-lm4tc\" (UID: \"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" Sep 07 11:01:19.445138 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.444998 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zp97j\" (UID: \"f3556cb2-da10-4588-bd2d-016629278608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:19.445138 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.445121 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgcv\" (UniqueName: \"kubernetes.io/projected/f3556cb2-da10-4588-bd2d-016629278608-kube-api-access-kjgcv\") pod \"cluster-samples-operator-7cdcfcf76c-zp97j\" (UID: \"f3556cb2-da10-4588-bd2d-016629278608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:19.445257 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.445147 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d-config\") pod \"service-ca-operator-859f94c4b-lm4tc\" (UID: \"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" Sep 07 11:01:19.546450 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.546367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d-serving-cert\") pod \"service-ca-operator-859f94c4b-lm4tc\" (UID: \"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" Sep 07 11:01:19.546450 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.546404 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgj2z\" (UniqueName: \"kubernetes.io/projected/ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d-kube-api-access-xgj2z\") pod \"service-ca-operator-859f94c4b-lm4tc\" (UID: \"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" Sep 07 11:01:19.546450 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.546426 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zp97j\" (UID: \"f3556cb2-da10-4588-bd2d-016629278608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:19.546723 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:19.546527 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 11:01:19.546723 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:19.546613 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls podName:f3556cb2-da10-4588-bd2d-016629278608 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:20.046595259 +0000 UTC m=+52.520951616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-zp97j" (UID: "f3556cb2-da10-4588-bd2d-016629278608") : secret "samples-operator-tls" not found Sep 07 11:01:19.546723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.546525 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgcv\" (UniqueName: \"kubernetes.io/projected/f3556cb2-da10-4588-bd2d-016629278608-kube-api-access-kjgcv\") pod \"cluster-samples-operator-7cdcfcf76c-zp97j\" (UID: \"f3556cb2-da10-4588-bd2d-016629278608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:19.546723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.546645 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f55c689e-68fb-4fbb-914f-dae8863f36cc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kv4fh\" (UID: \"f55c689e-68fb-4fbb-914f-dae8863f36cc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" Sep 07 11:01:19.546723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.546673 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d-config\") pod \"service-ca-operator-859f94c4b-lm4tc\" (UID: \"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" Sep 07 11:01:19.546723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.546701 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f55c689e-68fb-4fbb-914f-dae8863f36cc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kv4fh\" (UID: \"f55c689e-68fb-4fbb-914f-dae8863f36cc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" Sep 07 11:01:19.546971 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.546739 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr9hq\" (UniqueName: \"kubernetes.io/projected/f55c689e-68fb-4fbb-914f-dae8863f36cc-kube-api-access-tr9hq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kv4fh\" (UID: \"f55c689e-68fb-4fbb-914f-dae8863f36cc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" Sep 07 11:01:19.546971 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.546765 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98tt7\" (UniqueName: \"kubernetes.io/projected/24661e4f-4c12-4393-8f27-a47096220e14-kube-api-access-98tt7\") pod \"volume-data-source-validator-695c895bd6-2nrt8\" (UID: \"24661e4f-4c12-4393-8f27-a47096220e14\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2nrt8" Sep 07 11:01:19.547171 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.547138 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d-config\") pod \"service-ca-operator-859f94c4b-lm4tc\" (UID: \"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" Sep 07 11:01:19.550177 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.550158 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d-serving-cert\") pod \"service-ca-operator-859f94c4b-lm4tc\" (UID: \"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" Sep 07 11:01:19.554763 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.554738 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgj2z\" (UniqueName: \"kubernetes.io/projected/ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d-kube-api-access-xgj2z\") pod \"service-ca-operator-859f94c4b-lm4tc\" (UID: \"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" Sep 07 11:01:19.554845 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.554821 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgcv\" (UniqueName: \"kubernetes.io/projected/f3556cb2-da10-4588-bd2d-016629278608-kube-api-access-kjgcv\") pod \"cluster-samples-operator-7cdcfcf76c-zp97j\" (UID: \"f3556cb2-da10-4588-bd2d-016629278608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:19.585582 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.585536 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-cfb82"] Sep 07 11:01:19.600158 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.600129 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" Sep 07 11:01:19.607706 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.607680 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-cfb82"] Sep 07 11:01:19.607836 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.607768 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.610414 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.610386 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-p94vz\"" Sep 07 11:01:19.610414 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.610392 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 07 11:01:19.610617 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.610394 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 07 11:01:19.610617 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.610392 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 07 11:01:19.610728 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.610716 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 07 11:01:19.617853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.617833 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 07 11:01:19.648141 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.648107 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr9hq\" (UniqueName: \"kubernetes.io/projected/f55c689e-68fb-4fbb-914f-dae8863f36cc-kube-api-access-tr9hq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kv4fh\" (UID: \"f55c689e-68fb-4fbb-914f-dae8863f36cc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" Sep 07 11:01:19.648141 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.648145 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98tt7\" (UniqueName: \"kubernetes.io/projected/24661e4f-4c12-4393-8f27-a47096220e14-kube-api-access-98tt7\") pod \"volume-data-source-validator-695c895bd6-2nrt8\" (UID: \"24661e4f-4c12-4393-8f27-a47096220e14\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2nrt8" Sep 07 11:01:19.648377 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.648242 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f55c689e-68fb-4fbb-914f-dae8863f36cc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kv4fh\" (UID: \"f55c689e-68fb-4fbb-914f-dae8863f36cc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" Sep 07 11:01:19.648377 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.648266 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f55c689e-68fb-4fbb-914f-dae8863f36cc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kv4fh\" (UID: \"f55c689e-68fb-4fbb-914f-dae8863f36cc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" Sep 07 11:01:19.648843 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.648818 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f55c689e-68fb-4fbb-914f-dae8863f36cc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kv4fh\" (UID: \"f55c689e-68fb-4fbb-914f-dae8863f36cc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" Sep 07 11:01:19.650851 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.650824 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f55c689e-68fb-4fbb-914f-dae8863f36cc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kv4fh\" (UID: \"f55c689e-68fb-4fbb-914f-dae8863f36cc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" Sep 07 11:01:19.655735 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.655703 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr9hq\" (UniqueName: \"kubernetes.io/projected/f55c689e-68fb-4fbb-914f-dae8863f36cc-kube-api-access-tr9hq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kv4fh\" (UID: \"f55c689e-68fb-4fbb-914f-dae8863f36cc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" Sep 07 11:01:19.655884 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.655869 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98tt7\" (UniqueName: \"kubernetes.io/projected/24661e4f-4c12-4393-8f27-a47096220e14-kube-api-access-98tt7\") pod \"volume-data-source-validator-695c895bd6-2nrt8\" (UID: \"24661e4f-4c12-4393-8f27-a47096220e14\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2nrt8" Sep 07 11:01:19.701818 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.701787 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2nrt8" Sep 07 11:01:19.717989 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.717956 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" Sep 07 11:01:19.748741 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.748713 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f368693e-275e-42a4-8bef-ec9a8d380236-config\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.748861 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.748747 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f368693e-275e-42a4-8bef-ec9a8d380236-trusted-ca\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.748861 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.748776 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f368693e-275e-42a4-8bef-ec9a8d380236-serving-cert\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.748945 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.748878 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxnqd\" (UniqueName: \"kubernetes.io/projected/f368693e-275e-42a4-8bef-ec9a8d380236-kube-api-access-pxnqd\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.850106 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.850069 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f368693e-275e-42a4-8bef-ec9a8d380236-serving-cert\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.850269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.850133 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxnqd\" (UniqueName: \"kubernetes.io/projected/f368693e-275e-42a4-8bef-ec9a8d380236-kube-api-access-pxnqd\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.850269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.850211 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f368693e-275e-42a4-8bef-ec9a8d380236-config\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.850269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.850234 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f368693e-275e-42a4-8bef-ec9a8d380236-trusted-ca\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.850845 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.850816 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f368693e-275e-42a4-8bef-ec9a8d380236-config\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.851459 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.851435 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f368693e-275e-42a4-8bef-ec9a8d380236-trusted-ca\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.852353 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.852331 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f368693e-275e-42a4-8bef-ec9a8d380236-serving-cert\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.857995 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.857977 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxnqd\" (UniqueName: \"kubernetes.io/projected/f368693e-275e-42a4-8bef-ec9a8d380236-kube-api-access-pxnqd\") pod \"console-operator-85b454874b-cfb82\" (UID: \"f368693e-275e-42a4-8bef-ec9a8d380236\") " pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.880490 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.880461 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc"] Sep 07 11:01:19.883230 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:01:19.883201 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1e08eb_3b12_4b62_ac0d_7e7c7b470b0d.slice/crio-7a039f589c6a219728db349ac05b8e70bb37d2559db62346766d4cacb1ac434c WatchSource:0}: Error finding container 7a039f589c6a219728db349ac05b8e70bb37d2559db62346766d4cacb1ac434c: Status 404 returned error can't find the container with id 7a039f589c6a219728db349ac05b8e70bb37d2559db62346766d4cacb1ac434c Sep 07 11:01:19.918906 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.918862 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:19.998225 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:19.998091 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2nrt8"] Sep 07 11:01:20.001136 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:01:20.001108 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24661e4f_4c12_4393_8f27_a47096220e14.slice/crio-4c0f576d9998ef522eda44d7f29a9a2799a43fe2fa1a47eb6d0772822efa337c WatchSource:0}: Error finding container 4c0f576d9998ef522eda44d7f29a9a2799a43fe2fa1a47eb6d0772822efa337c: Status 404 returned error can't find the container with id 4c0f576d9998ef522eda44d7f29a9a2799a43fe2fa1a47eb6d0772822efa337c Sep 07 11:01:20.029173 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:20.028283 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh"] Sep 07 11:01:20.032868 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:01:20.032842 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55c689e_68fb_4fbb_914f_dae8863f36cc.slice/crio-ed946bed0b096d99dd34e1898f9e49514201510893f20bd47895d51796bd005d WatchSource:0}: Error finding container ed946bed0b096d99dd34e1898f9e49514201510893f20bd47895d51796bd005d: Status 404 returned error can't find the container with id ed946bed0b096d99dd34e1898f9e49514201510893f20bd47895d51796bd005d Sep 07 11:01:20.052519 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:20.052486 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zp97j\" (UID: \"f3556cb2-da10-4588-bd2d-016629278608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:20.052661 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:20.052642 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 11:01:20.052718 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:20.052708 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls podName:f3556cb2-da10-4588-bd2d-016629278608 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:21.052690785 +0000 UTC m=+53.527047142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-zp97j" (UID: "f3556cb2-da10-4588-bd2d-016629278608") : secret "samples-operator-tls" not found Sep 07 11:01:20.212914 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:20.212844 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-cfb82"] Sep 07 11:01:20.215570 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:01:20.215531 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf368693e_275e_42a4_8bef_ec9a8d380236.slice/crio-162847fa02171b454c31ed066f56badd497b3a07e8a87ea18169bde0521ff988 WatchSource:0}: Error finding container 162847fa02171b454c31ed066f56badd497b3a07e8a87ea18169bde0521ff988: Status 404 returned error can't find the container with id 162847fa02171b454c31ed066f56badd497b3a07e8a87ea18169bde0521ff988 Sep 07 11:01:20.326922 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:20.326883 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2nrt8" event={"ID":"24661e4f-4c12-4393-8f27-a47096220e14","Type":"ContainerStarted","Data":"4c0f576d9998ef522eda44d7f29a9a2799a43fe2fa1a47eb6d0772822efa337c"} Sep 07 11:01:20.328019 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:20.327998 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" event={"ID":"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d","Type":"ContainerStarted","Data":"7a039f589c6a219728db349ac05b8e70bb37d2559db62346766d4cacb1ac434c"} Sep 07 11:01:20.329036 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:20.329014 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-cfb82" event={"ID":"f368693e-275e-42a4-8bef-ec9a8d380236","Type":"ContainerStarted","Data":"162847fa02171b454c31ed066f56badd497b3a07e8a87ea18169bde0521ff988"} Sep 07 11:01:20.329987 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:20.329966 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" event={"ID":"f55c689e-68fb-4fbb-914f-dae8863f36cc","Type":"ContainerStarted","Data":"ed946bed0b096d99dd34e1898f9e49514201510893f20bd47895d51796bd005d"} Sep 07 11:01:21.062936 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:21.062870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zp97j\" (UID: \"f3556cb2-da10-4588-bd2d-016629278608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:21.063119 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:21.062979 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 11:01:21.063119 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:21.063046 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls podName:f3556cb2-da10-4588-bd2d-016629278608 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:23.063028586 +0000 UTC m=+55.537384947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-zp97j" (UID: "f3556cb2-da10-4588-bd2d-016629278608") : secret "samples-operator-tls" not found Sep 07 11:01:21.335333 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:21.335237 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2nrt8" event={"ID":"24661e4f-4c12-4393-8f27-a47096220e14","Type":"ContainerStarted","Data":"4c15c40a913e0fa88223acb9a9bb00c83bb28585896504498ea666845b6c612c"} Sep 07 11:01:21.352931 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:21.352525 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2nrt8" podStartSLOduration=1.183351419 podStartE2EDuration="2.352504415s" totalImagesPullingTime="1.169152996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:01:21.350503213 +0000 UTC m=+53.824859591" watchObservedRunningTime="2026-09-07 11:01:21.352504415 +0000 UTC m=+53.826860788" Sep 07 11:01:23.079537 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:23.079497 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zp97j\" (UID: \"f3556cb2-da10-4588-bd2d-016629278608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:23.079943 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:23.079633 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 11:01:23.079943 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:23.079672 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls podName:f3556cb2-da10-4588-bd2d-016629278608 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:27.079661533 +0000 UTC m=+59.554017889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-zp97j" (UID: "f3556cb2-da10-4588-bd2d-016629278608") : secret "samples-operator-tls" not found Sep 07 11:01:23.342969 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:23.342887 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/0.log" Sep 07 11:01:23.342969 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:23.342937 2571 generic.go:356] "Generic (PLEG): container finished" podID="f368693e-275e-42a4-8bef-ec9a8d380236" containerID="837a0691e2b7606e299a11d947801fcc38959ecfaddf5419d77ae40d98444ce4" exitCode=255 Sep 07 11:01:23.343183 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:23.343026 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-cfb82" event={"ID":"f368693e-275e-42a4-8bef-ec9a8d380236","Type":"ContainerDied","Data":"837a0691e2b7606e299a11d947801fcc38959ecfaddf5419d77ae40d98444ce4"} Sep 07 11:01:23.343258 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:23.343238 2571 scope.go:117] "RemoveContainer" containerID="837a0691e2b7606e299a11d947801fcc38959ecfaddf5419d77ae40d98444ce4" Sep 07 11:01:23.344704 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:23.344675 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" event={"ID":"f55c689e-68fb-4fbb-914f-dae8863f36cc","Type":"ContainerStarted","Data":"5be70cc8e18087858d2e43a6d63b0efa156ae084c288913d159eb319d44c1d76"} Sep 07 11:01:23.346290 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:23.346268 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" event={"ID":"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d","Type":"ContainerStarted","Data":"47047dc7910e50b64ad6768ce62b5f3ef52a71728ba5c9f993d497ee972f3f04"} Sep 07 11:01:23.371444 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:23.371389 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" podStartSLOduration=1.184223735 podStartE2EDuration="4.371370325s" totalImagesPullingTime="3.18714659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:01:23.370748697 +0000 UTC m=+55.845105078" watchObservedRunningTime="2026-09-07 11:01:23.371370325 +0000 UTC m=+55.845726704" Sep 07 11:01:23.386013 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:23.385960 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" podStartSLOduration=1.343192394 podStartE2EDuration="4.385939471s" totalImagesPullingTime="3.042747077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:01:23.38502364 +0000 UTC m=+55.859380018" watchObservedRunningTime="2026-09-07 11:01:23.385939471 +0000 UTC m=+55.860295845" Sep 07 11:01:24.350971 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:24.350886 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/1.log" Sep 07 11:01:24.351694 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:24.351668 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/0.log" Sep 07 11:01:24.351818 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:24.351714 2571 generic.go:356] "Generic (PLEG): container finished" podID="f368693e-275e-42a4-8bef-ec9a8d380236" containerID="1ba23ea2bb071e2fcde0027808477727b5318ae0fa7dedd091b278654a11acd6" exitCode=255 Sep 07 11:01:24.351818 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:24.351794 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-cfb82" event={"ID":"f368693e-275e-42a4-8bef-ec9a8d380236","Type":"ContainerDied","Data":"1ba23ea2bb071e2fcde0027808477727b5318ae0fa7dedd091b278654a11acd6"} Sep 07 11:01:24.351911 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:24.351855 2571 scope.go:117] "RemoveContainer" containerID="837a0691e2b7606e299a11d947801fcc38959ecfaddf5419d77ae40d98444ce4" Sep 07 11:01:24.352966 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:24.352949 2571 scope.go:117] "RemoveContainer" containerID="1ba23ea2bb071e2fcde0027808477727b5318ae0fa7dedd091b278654a11acd6" Sep 07 11:01:24.353459 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:24.353190 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-cfb82_openshift-console-operator(f368693e-275e-42a4-8bef-ec9a8d380236)\"" pod="openshift-console-operator/console-operator-85b454874b-cfb82" podUID="f368693e-275e-42a4-8bef-ec9a8d380236" Sep 07 11:01:25.234825 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.234790 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zkccb"] Sep 07 11:01:25.239156 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.239135 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.241454 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.241434 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 07 11:01:25.241736 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.241721 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 07 11:01:25.241857 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.241841 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 07 11:01:25.241994 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.241981 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 07 11:01:25.242781 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.242766 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4j64r\"" Sep 07 11:01:25.253584 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.253541 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zkccb"] Sep 07 11:01:25.297378 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.297331 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.297587 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.297388 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc9d7764-52d5-496a-80e5-8802db0a5d5c-crio-socket\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.297587 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.297483 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8qc7\" (UniqueName: \"kubernetes.io/projected/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-api-access-b8qc7\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.297587 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.297529 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc9d7764-52d5-496a-80e5-8802db0a5d5c-data-volume\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.297710 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.297587 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.355842 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.355812 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/1.log" Sep 07 11:01:25.356215 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.356143 2571 scope.go:117] "RemoveContainer" containerID="1ba23ea2bb071e2fcde0027808477727b5318ae0fa7dedd091b278654a11acd6" Sep 07 11:01:25.356296 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:25.356277 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-cfb82_openshift-console-operator(f368693e-275e-42a4-8bef-ec9a8d380236)\"" pod="openshift-console-operator/console-operator-85b454874b-cfb82" podUID="f368693e-275e-42a4-8bef-ec9a8d380236" Sep 07 11:01:25.398345 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.398295 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8qc7\" (UniqueName: \"kubernetes.io/projected/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-api-access-b8qc7\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.398572 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.398359 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc9d7764-52d5-496a-80e5-8802db0a5d5c-data-volume\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.398572 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.398407 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.398572 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.398465 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.398572 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.398493 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc9d7764-52d5-496a-80e5-8802db0a5d5c-crio-socket\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.398819 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:25.398600 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 11:01:25.398819 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:25.398617 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 11:01:25.398819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.398632 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc9d7764-52d5-496a-80e5-8802db0a5d5c-crio-socket\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.398819 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:25.398686 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls podName:dc9d7764-52d5-496a-80e5-8802db0a5d5c nodeName:}" failed. No retries permitted until 2026-09-07 11:01:25.898664637 +0000 UTC m=+58.373021010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkccb" (UID: "dc9d7764-52d5-496a-80e5-8802db0a5d5c") : secret "insights-runtime-extractor-tls" not found Sep 07 11:01:25.398819 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:25.398709 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm podName:dc9d7764-52d5-496a-80e5-8802db0a5d5c nodeName:}" failed. No retries permitted until 2026-09-07 11:01:25.898698331 +0000 UTC m=+58.373054693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zkccb" (UID: "dc9d7764-52d5-496a-80e5-8802db0a5d5c") : configmap "kube-rbac-proxy" not found Sep 07 11:01:25.398819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.398737 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc9d7764-52d5-496a-80e5-8802db0a5d5c-data-volume\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.406797 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.406768 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8qc7\" (UniqueName: \"kubernetes.io/projected/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-api-access-b8qc7\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.903393 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.903356 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.903628 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:25.903430 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:25.903628 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:25.903507 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 11:01:25.903628 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:25.903514 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 11:01:25.903628 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:25.903622 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm podName:dc9d7764-52d5-496a-80e5-8802db0a5d5c nodeName:}" failed. No retries permitted until 2026-09-07 11:01:26.903600807 +0000 UTC m=+59.377957187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zkccb" (UID: "dc9d7764-52d5-496a-80e5-8802db0a5d5c") : configmap "kube-rbac-proxy" not found Sep 07 11:01:25.903850 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:25.903678 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls podName:dc9d7764-52d5-496a-80e5-8802db0a5d5c nodeName:}" failed. No retries permitted until 2026-09-07 11:01:26.903665239 +0000 UTC m=+59.378021600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkccb" (UID: "dc9d7764-52d5-496a-80e5-8802db0a5d5c") : secret "insights-runtime-extractor-tls" not found Sep 07 11:01:26.589097 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:26.589069 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4fw4g_c839fa80-c4fe-4444-b4d3-4de7cef50ec6/dns-node-resolver/0.log" Sep 07 11:01:26.912810 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:26.912717 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:26.912945 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:26.912821 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:26.912945 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:26.912841 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 11:01:26.912945 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:26.912901 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm podName:dc9d7764-52d5-496a-80e5-8802db0a5d5c nodeName:}" failed. No retries permitted until 2026-09-07 11:01:28.912887654 +0000 UTC m=+61.387244044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zkccb" (UID: "dc9d7764-52d5-496a-80e5-8802db0a5d5c") : configmap "kube-rbac-proxy" not found Sep 07 11:01:26.912945 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:26.912935 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 11:01:26.913086 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:26.912981 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls podName:dc9d7764-52d5-496a-80e5-8802db0a5d5c nodeName:}" failed. No retries permitted until 2026-09-07 11:01:28.912969618 +0000 UTC m=+61.387325991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkccb" (UID: "dc9d7764-52d5-496a-80e5-8802db0a5d5c") : secret "insights-runtime-extractor-tls" not found Sep 07 11:01:27.114399 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:27.114353 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zp97j\" (UID: \"f3556cb2-da10-4588-bd2d-016629278608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:27.114596 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:27.114512 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 11:01:27.114651 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:27.114603 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls podName:f3556cb2-da10-4588-bd2d-016629278608 nodeName:}" failed. No retries permitted until 2026-09-07 11:01:35.114588824 +0000 UTC m=+67.588945185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-zp97j" (UID: "f3556cb2-da10-4588-bd2d-016629278608") : secret "samples-operator-tls" not found Sep 07 11:01:27.991447 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:27.991420 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r5q7t_973302dc-6097-4017-b889-c7bcd790d603/node-ca/0.log" Sep 07 11:01:28.272717 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:28.272685 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rlg9h" Sep 07 11:01:28.590325 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:28.590240 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sjcrj_e23c9000-ee54-41ff-8409-5da16eed5f60/serve-healthcheck-canary/0.log" Sep 07 11:01:28.930416 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:28.930316 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:28.930676 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:28.930421 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:28.930676 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:28.930458 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 11:01:28.930676 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:28.930542 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm podName:dc9d7764-52d5-496a-80e5-8802db0a5d5c nodeName:}" failed. No retries permitted until 2026-09-07 11:01:32.930527849 +0000 UTC m=+65.404884218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zkccb" (UID: "dc9d7764-52d5-496a-80e5-8802db0a5d5c") : configmap "kube-rbac-proxy" not found Sep 07 11:01:28.930676 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:28.930580 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 11:01:28.930676 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:28.930650 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls podName:dc9d7764-52d5-496a-80e5-8802db0a5d5c nodeName:}" failed. No retries permitted until 2026-09-07 11:01:32.930636776 +0000 UTC m=+65.404993138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkccb" (UID: "dc9d7764-52d5-496a-80e5-8802db0a5d5c") : secret "insights-runtime-extractor-tls" not found Sep 07 11:01:29.919367 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:29.919325 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:29.919767 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:29.919385 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:29.919767 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:29.919717 2571 scope.go:117] "RemoveContainer" containerID="1ba23ea2bb071e2fcde0027808477727b5318ae0fa7dedd091b278654a11acd6" Sep 07 11:01:29.919879 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:29.919859 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-cfb82_openshift-console-operator(f368693e-275e-42a4-8bef-ec9a8d380236)\"" pod="openshift-console-operator/console-operator-85b454874b-cfb82" podUID="f368693e-275e-42a4-8bef-ec9a8d380236" Sep 07 11:01:32.963244 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:32.963208 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:32.963634 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:32.963312 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:32.963634 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:32.963328 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 11:01:32.963634 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:32.963394 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm podName:dc9d7764-52d5-496a-80e5-8802db0a5d5c nodeName:}" failed. No retries permitted until 2026-09-07 11:01:40.963379482 +0000 UTC m=+73.437735885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zkccb" (UID: "dc9d7764-52d5-496a-80e5-8802db0a5d5c") : configmap "kube-rbac-proxy" not found Sep 07 11:01:32.963634 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:32.963409 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 11:01:32.963634 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:32.963464 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls podName:dc9d7764-52d5-496a-80e5-8802db0a5d5c nodeName:}" failed. No retries permitted until 2026-09-07 11:01:40.963449499 +0000 UTC m=+73.437805874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkccb" (UID: "dc9d7764-52d5-496a-80e5-8802db0a5d5c") : secret "insights-runtime-extractor-tls" not found Sep 07 11:01:33.770751 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:33.770714 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:01:33.770946 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:33.770800 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:01:33.773438 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:33.773416 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 11:01:33.773525 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:33.773505 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 11:01:33.781532 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:33.781511 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 07 11:01:33.781600 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:33.781589 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs podName:cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0 nodeName:}" failed. No retries permitted until 2026-09-07 11:02:37.781569081 +0000 UTC m=+130.255925442 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs") pod "network-metrics-daemon-6jc59" (UID: "cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0") : secret "metrics-daemon-secret" not found Sep 07 11:01:33.783626 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:33.783599 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fbcd4d6-16cc-4ba2-933f-6938932d59a3-original-pull-secret\") pod \"global-pull-secret-syncer-4vbhj\" (UID: \"7fbcd4d6-16cc-4ba2-933f-6938932d59a3\") " pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:01:33.838866 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:33.838829 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vbhj" Sep 07 11:01:33.871860 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:33.871826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwrm\" (UniqueName: \"kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm\") pod \"network-check-target-qz5nc\" (UID: \"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084\") " pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:01:33.874710 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:33.874685 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 07 11:01:33.885660 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:33.885637 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 07 11:01:33.896128 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:33.896100 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwrm\" (UniqueName: \"kubernetes.io/projected/fc08ced6-7a36-4ef2-b8e2-7cd3eb377084-kube-api-access-gzwrm\") pod \"network-check-target-qz5nc\" (UID: \"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084\") " pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:01:34.133570 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:34.133473 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4v4fp\"" Sep 07 11:01:34.141580 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:34.141543 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:01:34.148742 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:34.148713 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4vbhj"] Sep 07 11:01:34.152041 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:01:34.152019 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbcd4d6_16cc_4ba2_933f_6938932d59a3.slice/crio-685de327b53bfa7ac4120fd11361f6946a86c97da4596772faadb77457d06ea1 WatchSource:0}: Error finding container 685de327b53bfa7ac4120fd11361f6946a86c97da4596772faadb77457d06ea1: Status 404 returned error can't find the container with id 685de327b53bfa7ac4120fd11361f6946a86c97da4596772faadb77457d06ea1 Sep 07 11:01:34.274468 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:34.274428 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:01:34.274646 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:34.274607 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 11:01:34.274646 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:34.274625 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd4d7ff94-btjx6: secret "image-registry-tls" not found Sep 07 11:01:34.274726 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:34.274680 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls podName:5a201c33-d8a9-4792-9add-57056228abf6 nodeName:}" failed. No retries permitted until 2026-09-07 11:02:06.274663415 +0000 UTC m=+98.749020018 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls") pod "image-registry-dd4d7ff94-btjx6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6") : secret "image-registry-tls" not found Sep 07 11:01:34.375368 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:34.375329 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:01:34.375570 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:34.375475 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 11:01:34.375570 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:34.375537 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls podName:cd09d09c-a0d7-4104-8d79-746a7d0b8030 nodeName:}" failed. No retries permitted until 2026-09-07 11:02:06.375523515 +0000 UTC m=+98.849879871 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls") pod "dns-default-49lhf" (UID: "cd09d09c-a0d7-4104-8d79-746a7d0b8030") : secret "dns-default-metrics-tls" not found Sep 07 11:01:34.379455 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:34.379428 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4vbhj" event={"ID":"7fbcd4d6-16cc-4ba2-933f-6938932d59a3","Type":"ContainerStarted","Data":"685de327b53bfa7ac4120fd11361f6946a86c97da4596772faadb77457d06ea1"} Sep 07 11:01:34.437362 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:34.437289 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qz5nc"] Sep 07 11:01:34.440045 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:01:34.440018 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc08ced6_7a36_4ef2_b8e2_7cd3eb377084.slice/crio-e203f8f50619e074a9975f80627754cdcf3625a6a1fdeb41780796eb747500e3 WatchSource:0}: Error finding container e203f8f50619e074a9975f80627754cdcf3625a6a1fdeb41780796eb747500e3: Status 404 returned error can't find the container with id e203f8f50619e074a9975f80627754cdcf3625a6a1fdeb41780796eb747500e3 Sep 07 11:01:35.183088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:35.182949 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zp97j\" (UID: \"f3556cb2-da10-4588-bd2d-016629278608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:35.185925 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:35.185898 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3556cb2-da10-4588-bd2d-016629278608-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-zp97j\" (UID: \"f3556cb2-da10-4588-bd2d-016629278608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:35.210254 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:35.210221 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nqnjj\"" Sep 07 11:01:35.218374 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:35.218340 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" Sep 07 11:01:35.383945 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:35.383905 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qz5nc" event={"ID":"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084","Type":"ContainerStarted","Data":"e203f8f50619e074a9975f80627754cdcf3625a6a1fdeb41780796eb747500e3"} Sep 07 11:01:35.501391 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:35.501308 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j"] Sep 07 11:01:36.389106 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:36.389057 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" event={"ID":"f3556cb2-da10-4588-bd2d-016629278608","Type":"ContainerStarted","Data":"717e4244894fc32eca9b9f1094a888c535f5e322f398fce15f7d86df084c912e"} Sep 07 11:01:39.399914 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:39.399869 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qz5nc" event={"ID":"fc08ced6-7a36-4ef2-b8e2-7cd3eb377084","Type":"ContainerStarted","Data":"87e2570fe448cef970bebf131d6c10c664ca965f6483824a1b33087923ce029b"} Sep 07 11:01:39.400337 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:39.399965 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:01:39.401845 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:39.401815 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" event={"ID":"f3556cb2-da10-4588-bd2d-016629278608","Type":"ContainerStarted","Data":"19002ccc249196ab291758c5e33ac5394d5ad2ebccce1e2f54335099974ad0c9"} Sep 07 11:01:39.417233 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:39.417191 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qz5nc" podStartSLOduration=66.568785313 podStartE2EDuration="1m11.417176074s" totalImagesPullingTime="4.848390761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:01:39.417144394 +0000 UTC m=+71.891500772" watchObservedRunningTime="2026-09-07 11:01:39.417176074 +0000 UTC m=+71.891532483" Sep 07 11:01:40.118748 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:40.118717 2571 scope.go:117] "RemoveContainer" containerID="1ba23ea2bb071e2fcde0027808477727b5318ae0fa7dedd091b278654a11acd6" Sep 07 11:01:40.407177 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:40.407094 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4vbhj" event={"ID":"7fbcd4d6-16cc-4ba2-933f-6938932d59a3","Type":"ContainerStarted","Data":"7df01422e7c3313593ffb16aa5d3d636120cf910f6ddbf8620e7e898621dc79c"} Sep 07 11:01:40.408682 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:40.408660 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:01:40.409062 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:40.409045 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/1.log" Sep 07 11:01:40.409115 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:40.409078 2571 generic.go:356] "Generic (PLEG): container finished" podID="f368693e-275e-42a4-8bef-ec9a8d380236" containerID="37c045189c30877e4cd5a104257b92559c6fb1d585f30677c4a95ed135d86047" exitCode=255 Sep 07 11:01:40.409151 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:40.409139 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-cfb82" event={"ID":"f368693e-275e-42a4-8bef-ec9a8d380236","Type":"ContainerDied","Data":"37c045189c30877e4cd5a104257b92559c6fb1d585f30677c4a95ed135d86047"} Sep 07 11:01:40.409195 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:40.409177 2571 scope.go:117] "RemoveContainer" containerID="1ba23ea2bb071e2fcde0027808477727b5318ae0fa7dedd091b278654a11acd6" Sep 07 11:01:40.409474 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:40.409454 2571 scope.go:117] "RemoveContainer" containerID="37c045189c30877e4cd5a104257b92559c6fb1d585f30677c4a95ed135d86047" Sep 07 11:01:40.409673 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:40.409650 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-cfb82_openshift-console-operator(f368693e-275e-42a4-8bef-ec9a8d380236)\"" pod="openshift-console-operator/console-operator-85b454874b-cfb82" podUID="f368693e-275e-42a4-8bef-ec9a8d380236" Sep 07 11:01:40.410973 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:40.410944 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" event={"ID":"f3556cb2-da10-4588-bd2d-016629278608","Type":"ContainerStarted","Data":"1e9a499a537e2564b3e47dfc15cdb07c2a8d2c51326098b3fbdeee3154164da8"} Sep 07 11:01:40.425367 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:40.424376 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4vbhj" podStartSLOduration=67.277325069 podStartE2EDuration="1m12.424347087s" totalImagesPullingTime="5.147022018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:01:40.421364601 +0000 UTC m=+72.895720985" watchObservedRunningTime="2026-09-07 11:01:40.424347087 +0000 UTC m=+72.898703463" Sep 07 11:01:40.456170 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:40.456119 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-zp97j" podStartSLOduration=17.706320435 podStartE2EDuration="21.456105527s" totalImagesPullingTime="3.749785092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:01:40.455196091 +0000 UTC m=+72.929552470" watchObservedRunningTime="2026-09-07 11:01:40.456105527 +0000 UTC m=+72.930461905" Sep 07 11:01:41.040223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:41.040165 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:41.040415 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:41.040266 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:41.040415 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:41.040316 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 11:01:41.040415 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:41.040392 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls podName:dc9d7764-52d5-496a-80e5-8802db0a5d5c nodeName:}" failed. No retries permitted until 2026-09-07 11:01:57.040375836 +0000 UTC m=+89.514732192 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zkccb" (UID: "dc9d7764-52d5-496a-80e5-8802db0a5d5c") : secret "insights-runtime-extractor-tls" not found Sep 07 11:01:41.040809 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:41.040791 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc9d7764-52d5-496a-80e5-8802db0a5d5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:41.415425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:41.415345 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:01:46.159356 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.159320 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9"] Sep 07 11:01:46.165191 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.165161 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9" Sep 07 11:01:46.169071 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.169045 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ngw95\"" Sep 07 11:01:46.169163 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.169074 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 07 11:01:46.169163 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.169076 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 07 11:01:46.169163 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.169135 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 07 11:01:46.169163 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.169148 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 07 11:01:46.173466 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.173442 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9"] Sep 07 11:01:46.283983 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.283955 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbc95311-c31f-4fcd-b171-56ee6e4ae566-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76bf764dd6-tmfg9\" (UID: \"dbc95311-c31f-4fcd-b171-56ee6e4ae566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9" Sep 07 11:01:46.284129 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.284041 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf4zw\" (UniqueName: \"kubernetes.io/projected/dbc95311-c31f-4fcd-b171-56ee6e4ae566-kube-api-access-bf4zw\") pod \"managed-serviceaccount-addon-agent-76bf764dd6-tmfg9\" (UID: \"dbc95311-c31f-4fcd-b171-56ee6e4ae566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9" Sep 07 11:01:46.384820 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.384789 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbc95311-c31f-4fcd-b171-56ee6e4ae566-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76bf764dd6-tmfg9\" (UID: \"dbc95311-c31f-4fcd-b171-56ee6e4ae566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9" Sep 07 11:01:46.384955 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.384868 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf4zw\" (UniqueName: \"kubernetes.io/projected/dbc95311-c31f-4fcd-b171-56ee6e4ae566-kube-api-access-bf4zw\") pod \"managed-serviceaccount-addon-agent-76bf764dd6-tmfg9\" (UID: \"dbc95311-c31f-4fcd-b171-56ee6e4ae566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9" Sep 07 11:01:46.387152 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.387126 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbc95311-c31f-4fcd-b171-56ee6e4ae566-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76bf764dd6-tmfg9\" (UID: \"dbc95311-c31f-4fcd-b171-56ee6e4ae566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9" Sep 07 11:01:46.395293 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.395271 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf4zw\" (UniqueName: \"kubernetes.io/projected/dbc95311-c31f-4fcd-b171-56ee6e4ae566-kube-api-access-bf4zw\") pod \"managed-serviceaccount-addon-agent-76bf764dd6-tmfg9\" (UID: \"dbc95311-c31f-4fcd-b171-56ee6e4ae566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9" Sep 07 11:01:46.484774 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.484711 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9" Sep 07 11:01:46.751066 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:46.750998 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9"] Sep 07 11:01:46.754350 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:01:46.754324 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbc95311_c31f_4fcd_b171_56ee6e4ae566.slice/crio-32b73a078b81ede191aaddcd75ff307dd5dceb766a6903607a6f6d584eced993 WatchSource:0}: Error finding container 32b73a078b81ede191aaddcd75ff307dd5dceb766a6903607a6f6d584eced993: Status 404 returned error can't find the container with id 32b73a078b81ede191aaddcd75ff307dd5dceb766a6903607a6f6d584eced993 Sep 07 11:01:47.437146 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:47.437103 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9" event={"ID":"dbc95311-c31f-4fcd-b171-56ee6e4ae566","Type":"ContainerStarted","Data":"32b73a078b81ede191aaddcd75ff307dd5dceb766a6903607a6f6d584eced993"} Sep 07 11:01:49.447398 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:49.447309 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9" event={"ID":"dbc95311-c31f-4fcd-b171-56ee6e4ae566","Type":"ContainerStarted","Data":"7f6b53c2a839fa3eacf96e1d37cbf94996ce4b378bf078f9c5eaf52277f5215b"} Sep 07 11:01:49.461020 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:49.460973 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76bf764dd6-tmfg9" podStartSLOduration=1.166711657 podStartE2EDuration="3.460958843s" totalImagesPullingTime="2.294247186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:01:49.460166461 +0000 UTC m=+81.934522839" watchObservedRunningTime="2026-09-07 11:01:49.460958843 +0000 UTC m=+81.935315220" Sep 07 11:01:49.919732 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:49.919689 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:49.919976 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:49.919779 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:01:49.920072 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:49.920061 2571 scope.go:117] "RemoveContainer" containerID="37c045189c30877e4cd5a104257b92559c6fb1d585f30677c4a95ed135d86047" Sep 07 11:01:49.920213 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:01:49.920195 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-cfb82_openshift-console-operator(f368693e-275e-42a4-8bef-ec9a8d380236)\"" pod="openshift-console-operator/console-operator-85b454874b-cfb82" podUID="f368693e-275e-42a4-8bef-ec9a8d380236" Sep 07 11:01:57.067102 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:57.066973 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:57.069265 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:57.069240 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc9d7764-52d5-496a-80e5-8802db0a5d5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zkccb\" (UID: \"dc9d7764-52d5-496a-80e5-8802db0a5d5c\") " pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:57.350164 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:57.350100 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4j64r\"" Sep 07 11:01:57.358241 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:57.358221 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zkccb" Sep 07 11:01:57.659540 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:57.659512 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zkccb"] Sep 07 11:01:57.661735 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:01:57.661707 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9d7764_52d5_496a_80e5_8802db0a5d5c.slice/crio-9513e7d6f196085511dedeb6c3755a7a12618e749fc19320b15a686146f12863 WatchSource:0}: Error finding container 9513e7d6f196085511dedeb6c3755a7a12618e749fc19320b15a686146f12863: Status 404 returned error can't find the container with id 9513e7d6f196085511dedeb6c3755a7a12618e749fc19320b15a686146f12863 Sep 07 11:01:58.476181 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:58.476150 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zkccb" event={"ID":"dc9d7764-52d5-496a-80e5-8802db0a5d5c","Type":"ContainerStarted","Data":"3b35b2cca428a66c66f614319aa1d087e7487f9918d6e50fd2332058ce24b3ee"} Sep 07 11:01:58.476522 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:58.476204 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zkccb" event={"ID":"dc9d7764-52d5-496a-80e5-8802db0a5d5c","Type":"ContainerStarted","Data":"9513e7d6f196085511dedeb6c3755a7a12618e749fc19320b15a686146f12863"} Sep 07 11:01:59.482263 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:01:59.482215 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zkccb" event={"ID":"dc9d7764-52d5-496a-80e5-8802db0a5d5c","Type":"ContainerStarted","Data":"3312ce297ebe9048f9f958d2c478ad7e2839187fa15b3a275a804d15a2a05673"} Sep 07 11:02:00.491577 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:00.491474 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zkccb" event={"ID":"dc9d7764-52d5-496a-80e5-8802db0a5d5c","Type":"ContainerStarted","Data":"94f7807860a61ef57421eff7e2853171f96a608b14a22d79b816e4c5fb05ddfc"} Sep 07 11:02:00.508446 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:00.508397 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zkccb" podStartSLOduration=33.275938442 podStartE2EDuration="35.508381813s" totalImagesPullingTime="2.232443371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:01:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:00.507524746 +0000 UTC m=+92.981881126" watchObservedRunningTime="2026-09-07 11:02:00.508381813 +0000 UTC m=+92.982738190" Sep 07 11:02:03.382786 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.382754 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g"] Sep 07 11:02:03.385830 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.385815 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.388281 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.388261 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 07 11:02:03.389644 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.389624 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-s4qzv\"" Sep 07 11:02:03.389644 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.389632 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 07 11:02:03.389819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.389670 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 07 11:02:03.389819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.389685 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 07 11:02:03.389819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.389721 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 07 11:02:03.399687 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.399667 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g"] Sep 07 11:02:03.402097 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.402079 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-svn6v"] Sep 07 11:02:03.405001 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.404984 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.407920 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.407887 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mz6tj\"" Sep 07 11:02:03.408022 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.407924 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 07 11:02:03.408022 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.407947 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 07 11:02:03.408022 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.407905 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 07 11:02:03.416149 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.416125 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-svn6v"] Sep 07 11:02:03.425894 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.425875 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p67qs"] Sep 07 11:02:03.428882 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.428868 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.431301 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.431277 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ltd7n\"" Sep 07 11:02:03.431389 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.431326 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 07 11:02:03.431451 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.431408 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 07 11:02:03.431578 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.431564 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 07 11:02:03.512496 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512471 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d734a792-d188-4c32-8919-122260339217-node-exporter-wtmp\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.512623 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70eb15f4-2ce4-4ceb-9113-203baff5b63c-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.512623 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512522 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgsd9\" (UniqueName: \"kubernetes.io/projected/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-api-access-kgsd9\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.512623 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512581 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.512623 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512621 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cskv2\" (UniqueName: \"kubernetes.io/projected/d734a792-d188-4c32-8919-122260339217-kube-api-access-cskv2\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.512785 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512641 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d734a792-d188-4c32-8919-122260339217-node-exporter-tls\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.512785 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512660 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mqc\" (UniqueName: \"kubernetes.io/projected/18ba05e2-441f-4905-a567-a93fccc511de-kube-api-access-77mqc\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.512785 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512682 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/70eb15f4-2ce4-4ceb-9113-203baff5b63c-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.512785 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512732 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d734a792-d188-4c32-8919-122260339217-sys\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.512785 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512748 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d734a792-d188-4c32-8919-122260339217-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.512785 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512773 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.512973 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512810 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18ba05e2-441f-4905-a567-a93fccc511de-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.512973 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512834 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d734a792-d188-4c32-8919-122260339217-root\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.512973 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512848 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d734a792-d188-4c32-8919-122260339217-metrics-client-ca\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.512973 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512891 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18ba05e2-441f-4905-a567-a93fccc511de-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.512973 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512932 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d734a792-d188-4c32-8919-122260339217-node-exporter-accelerators-collector-config\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.512973 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512953 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18ba05e2-441f-4905-a567-a93fccc511de-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.512973 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512967 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d734a792-d188-4c32-8919-122260339217-node-exporter-textfile\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.513171 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.512983 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.613973 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.613942 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70eb15f4-2ce4-4ceb-9113-203baff5b63c-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.614098 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.613984 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgsd9\" (UniqueName: \"kubernetes.io/projected/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-api-access-kgsd9\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.614098 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614067 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.614214 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614115 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cskv2\" (UniqueName: \"kubernetes.io/projected/d734a792-d188-4c32-8919-122260339217-kube-api-access-cskv2\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.614262 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614234 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d734a792-d188-4c32-8919-122260339217-node-exporter-tls\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.614311 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614289 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mqc\" (UniqueName: \"kubernetes.io/projected/18ba05e2-441f-4905-a567-a93fccc511de-kube-api-access-77mqc\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.614362 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614322 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/70eb15f4-2ce4-4ceb-9113-203baff5b63c-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.614412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614368 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d734a792-d188-4c32-8919-122260339217-sys\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.614412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614396 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d734a792-d188-4c32-8919-122260339217-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.614511 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:02:03.614415 2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 07 11:02:03.614511 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614441 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.614511 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:02:03.614474 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d734a792-d188-4c32-8919-122260339217-node-exporter-tls podName:d734a792-d188-4c32-8919-122260339217 nodeName:}" failed. No retries permitted until 2026-09-07 11:02:04.114455778 +0000 UTC m=+96.588812140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d734a792-d188-4c32-8919-122260339217-node-exporter-tls") pod "node-exporter-p67qs" (UID: "d734a792-d188-4c32-8919-122260339217") : secret "node-exporter-tls" not found Sep 07 11:02:03.614511 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614494 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18ba05e2-441f-4905-a567-a93fccc511de-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:02:03.614530 2571 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614538 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d734a792-d188-4c32-8919-122260339217-root\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614545 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d734a792-d188-4c32-8919-122260339217-sys\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:02:03.614592 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-tls podName:70eb15f4-2ce4-4ceb-9113-203baff5b63c nodeName:}" failed. No retries permitted until 2026-09-07 11:02:04.114576549 +0000 UTC m=+96.588932964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-svn6v" (UID: "70eb15f4-2ce4-4ceb-9113-203baff5b63c") : secret "kube-state-metrics-tls" not found Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614618 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d734a792-d188-4c32-8919-122260339217-metrics-client-ca\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614651 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18ba05e2-441f-4905-a567-a93fccc511de-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614665 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d734a792-d188-4c32-8919-122260339217-root\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614712 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d734a792-d188-4c32-8919-122260339217-node-exporter-accelerators-collector-config\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614749 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18ba05e2-441f-4905-a567-a93fccc511de-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614788 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d734a792-d188-4c32-8919-122260339217-node-exporter-textfile\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614823 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614837 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/70eb15f4-2ce4-4ceb-9113-203baff5b63c-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.615055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614874 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d734a792-d188-4c32-8919-122260339217-node-exporter-wtmp\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.615697 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.614908 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70eb15f4-2ce4-4ceb-9113-203baff5b63c-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.615778 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.615752 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d734a792-d188-4c32-8919-122260339217-node-exporter-wtmp\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.615778 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.615764 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d734a792-d188-4c32-8919-122260339217-node-exporter-textfile\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.618269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.618242 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.618463 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.618431 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18ba05e2-441f-4905-a567-a93fccc511de-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.618807 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.618752 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d734a792-d188-4c32-8919-122260339217-node-exporter-accelerators-collector-config\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.623184 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.618960 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d734a792-d188-4c32-8919-122260339217-metrics-client-ca\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.624905 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.624387 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18ba05e2-441f-4905-a567-a93fccc511de-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.625759 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.625402 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cskv2\" (UniqueName: \"kubernetes.io/projected/d734a792-d188-4c32-8919-122260339217-kube-api-access-cskv2\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.625857 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.625828 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgsd9\" (UniqueName: \"kubernetes.io/projected/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-api-access-kgsd9\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.627595 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.627568 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d734a792-d188-4c32-8919-122260339217-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:03.628056 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.628032 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:03.628194 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.628171 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18ba05e2-441f-4905-a567-a93fccc511de-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.628400 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.628383 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mqc\" (UniqueName: \"kubernetes.io/projected/18ba05e2-441f-4905-a567-a93fccc511de-kube-api-access-77mqc\") pod \"openshift-state-metrics-657fd6c54-2mz5g\" (UID: \"18ba05e2-441f-4905-a567-a93fccc511de\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.694590 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.694524 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" Sep 07 11:02:03.961721 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:03.961645 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g"] Sep 07 11:02:03.964234 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:02:03.964214 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ba05e2_441f_4905_a567_a93fccc511de.slice/crio-78c925094d27ca2b770ed18a759b688a21b18858fb26870d3378bfce6052c1cd WatchSource:0}: Error finding container 78c925094d27ca2b770ed18a759b688a21b18858fb26870d3378bfce6052c1cd: Status 404 returned error can't find the container with id 78c925094d27ca2b770ed18a759b688a21b18858fb26870d3378bfce6052c1cd Sep 07 11:02:04.118798 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:04.118760 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d734a792-d188-4c32-8919-122260339217-node-exporter-tls\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:04.118966 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:04.118813 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:04.121078 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:04.121052 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d734a792-d188-4c32-8919-122260339217-node-exporter-tls\") pod \"node-exporter-p67qs\" (UID: \"d734a792-d188-4c32-8919-122260339217\") " pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:04.121194 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:04.121097 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/70eb15f4-2ce4-4ceb-9113-203baff5b63c-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-svn6v\" (UID: \"70eb15f4-2ce4-4ceb-9113-203baff5b63c\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:04.312672 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:04.312644 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" Sep 07 11:02:04.336721 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:04.336687 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p67qs" Sep 07 11:02:04.347813 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:02:04.347784 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd734a792_d188_4c32_8919_122260339217.slice/crio-d9491d3351e5828dc2c635c7f69a523153a955ff53291f92bf8dd506c5d6e3f4 WatchSource:0}: Error finding container d9491d3351e5828dc2c635c7f69a523153a955ff53291f92bf8dd506c5d6e3f4: Status 404 returned error can't find the container with id d9491d3351e5828dc2c635c7f69a523153a955ff53291f92bf8dd506c5d6e3f4 Sep 07 11:02:04.506036 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:04.505998 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" event={"ID":"18ba05e2-441f-4905-a567-a93fccc511de","Type":"ContainerStarted","Data":"178944d57eaff157a0e940bbd84c574b90990c1a7accc0cd0879c4928ed5a79d"} Sep 07 11:02:04.506410 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:04.506059 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" event={"ID":"18ba05e2-441f-4905-a567-a93fccc511de","Type":"ContainerStarted","Data":"356f7c6abca6056d00b122806d0804afa9cc81841d71eccc2a245081e01a86cd"} Sep 07 11:02:04.506410 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:04.506080 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" event={"ID":"18ba05e2-441f-4905-a567-a93fccc511de","Type":"ContainerStarted","Data":"78c925094d27ca2b770ed18a759b688a21b18858fb26870d3378bfce6052c1cd"} Sep 07 11:02:04.507164 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:04.507141 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p67qs" event={"ID":"d734a792-d188-4c32-8919-122260339217","Type":"ContainerStarted","Data":"d9491d3351e5828dc2c635c7f69a523153a955ff53291f92bf8dd506c5d6e3f4"} Sep 07 11:02:04.586475 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:04.586285 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-svn6v"] Sep 07 11:02:04.588862 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:02:04.588839 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70eb15f4_2ce4_4ceb_9113_203baff5b63c.slice/crio-8c799870c4b4afa98d4bd62953cbc2ae3d53c769e8556849992874301aa84f70 WatchSource:0}: Error finding container 8c799870c4b4afa98d4bd62953cbc2ae3d53c769e8556849992874301aa84f70: Status 404 returned error can't find the container with id 8c799870c4b4afa98d4bd62953cbc2ae3d53c769e8556849992874301aa84f70 Sep 07 11:02:05.119193 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:05.119170 2571 scope.go:117] "RemoveContainer" containerID="37c045189c30877e4cd5a104257b92559c6fb1d585f30677c4a95ed135d86047" Sep 07 11:02:05.512671 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:05.512638 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" event={"ID":"18ba05e2-441f-4905-a567-a93fccc511de","Type":"ContainerStarted","Data":"b2d8f817fb523529b1dfe488fc2cdf6f2672269110da226062e5187093c34f86"} Sep 07 11:02:05.514902 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:05.514882 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:02:05.515001 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:05.514988 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-cfb82" event={"ID":"f368693e-275e-42a4-8bef-ec9a8d380236","Type":"ContainerStarted","Data":"b8fa348a92bb2e83ebfc9febfb18adffe61626592441bafae56cc9b5a8f96b8e"} Sep 07 11:02:05.515372 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:05.515346 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:02:05.516533 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:05.516483 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" event={"ID":"70eb15f4-2ce4-4ceb-9113-203baff5b63c","Type":"ContainerStarted","Data":"8c799870c4b4afa98d4bd62953cbc2ae3d53c769e8556849992874301aa84f70"} Sep 07 11:02:05.532522 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:05.532474 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2mz5g" podStartSLOduration=1.514769161 podStartE2EDuration="2.532456792s" totalImagesPullingTime="1.017687631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:05.531432303 +0000 UTC m=+98.005788682" watchObservedRunningTime="2026-09-07 11:02:05.532456792 +0000 UTC m=+98.006813169" Sep 07 11:02:05.549190 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:05.549138 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-cfb82" podStartSLOduration=43.69251752 podStartE2EDuration="46.549118131s" totalImagesPullingTime="2.856600611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:05.54790633 +0000 UTC m=+98.022262706" watchObservedRunningTime="2026-09-07 11:02:05.549118131 +0000 UTC m=+98.023474511" Sep 07 11:02:05.682787 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:05.682753 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-cfb82" Sep 07 11:02:06.338871 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.338826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:02:06.341922 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.341890 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") pod \"image-registry-dd4d7ff94-btjx6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:02:06.355291 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.355265 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rmbnm\"" Sep 07 11:02:06.363280 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.363256 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:02:06.440131 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.440101 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:02:06.442232 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.442209 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd09d09c-a0d7-4104-8d79-746a7d0b8030-metrics-tls\") pod \"dns-default-49lhf\" (UID: \"cd09d09c-a0d7-4104-8d79-746a7d0b8030\") " pod="openshift-dns/dns-default-49lhf" Sep 07 11:02:06.523973 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.523923 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" event={"ID":"70eb15f4-2ce4-4ceb-9113-203baff5b63c","Type":"ContainerStarted","Data":"ae3b9e71542ad4ac7992b442be8edefe6e2d182ca9d7359d5731770b361637e6"} Sep 07 11:02:06.524428 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.524002 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" event={"ID":"70eb15f4-2ce4-4ceb-9113-203baff5b63c","Type":"ContainerStarted","Data":"657ed79f6fd87c0356dc612e05a2ed7d4a5086a70b5815b64b7cd8c0047ed1dc"} Sep 07 11:02:06.524428 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.524030 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" event={"ID":"70eb15f4-2ce4-4ceb-9113-203baff5b63c","Type":"ContainerStarted","Data":"593f4ec25b22a664620c551ab0cfea41777f378fb04013ad5dd2bfbae3aec694"} Sep 07 11:02:06.525603 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.525571 2571 generic.go:356] "Generic (PLEG): container finished" podID="d734a792-d188-4c32-8919-122260339217" containerID="75c19fd13416b7ac6e4b04b02e468238de46a8ae622216e53b18cf481d512a93" exitCode=0 Sep 07 11:02:06.525711 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.525654 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p67qs" event={"ID":"d734a792-d188-4c32-8919-122260339217","Type":"ContainerDied","Data":"75c19fd13416b7ac6e4b04b02e468238de46a8ae622216e53b18cf481d512a93"} Sep 07 11:02:06.557990 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.557944 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-svn6v" podStartSLOduration=2.136266799 podStartE2EDuration="3.557925985s" totalImagesPullingTime="1.421659186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:06.556832315 +0000 UTC m=+99.031188706" watchObservedRunningTime="2026-09-07 11:02:06.557925985 +0000 UTC m=+99.032282365" Sep 07 11:02:06.636697 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.636668 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dd4d7ff94-btjx6"] Sep 07 11:02:06.639520 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:02:06.639495 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a201c33_d8a9_4792_9add_57056228abf6.slice/crio-8991d03184813c4ef24f9bd8a65a6687949c5d4d02b8645a13afd9a89afe1a0f WatchSource:0}: Error finding container 8991d03184813c4ef24f9bd8a65a6687949c5d4d02b8645a13afd9a89afe1a0f: Status 404 returned error can't find the container with id 8991d03184813c4ef24f9bd8a65a6687949c5d4d02b8645a13afd9a89afe1a0f Sep 07 11:02:06.668962 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.668942 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-czp2d\"" Sep 07 11:02:06.677359 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.677201 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-49lhf" Sep 07 11:02:06.946625 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:06.946534 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-49lhf"] Sep 07 11:02:06.950495 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:02:06.950462 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd09d09c_a0d7_4104_8d79_746a7d0b8030.slice/crio-06a1f272c53e2f670e6fab79ae28a6035174b4cbd69ba4e2ececeb629991b9a6 WatchSource:0}: Error finding container 06a1f272c53e2f670e6fab79ae28a6035174b4cbd69ba4e2ececeb629991b9a6: Status 404 returned error can't find the container with id 06a1f272c53e2f670e6fab79ae28a6035174b4cbd69ba4e2ececeb629991b9a6 Sep 07 11:02:07.537310 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.537268 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-49lhf" event={"ID":"cd09d09c-a0d7-4104-8d79-746a7d0b8030","Type":"ContainerStarted","Data":"06a1f272c53e2f670e6fab79ae28a6035174b4cbd69ba4e2ececeb629991b9a6"} Sep 07 11:02:07.540360 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.540330 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p67qs" event={"ID":"d734a792-d188-4c32-8919-122260339217","Type":"ContainerStarted","Data":"73d00938ca51b3cdd558fbcea44507bfc058dbb8fc8beac15d8c8813be910705"} Sep 07 11:02:07.540482 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.540378 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p67qs" event={"ID":"d734a792-d188-4c32-8919-122260339217","Type":"ContainerStarted","Data":"ed5efab9df1b635e3b2ec34888820fe9f269c36cfc9ec10e1a74a94841294e6a"} Sep 07 11:02:07.542080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.542045 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" event={"ID":"5a201c33-d8a9-4792-9add-57056228abf6","Type":"ContainerStarted","Data":"feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5"} Sep 07 11:02:07.542180 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.542095 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" event={"ID":"5a201c33-d8a9-4792-9add-57056228abf6","Type":"ContainerStarted","Data":"8991d03184813c4ef24f9bd8a65a6687949c5d4d02b8645a13afd9a89afe1a0f"} Sep 07 11:02:07.542533 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.542511 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:02:07.562003 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.561947 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p67qs" podStartSLOduration=3.396690709 podStartE2EDuration="4.56192796s" totalImagesPullingTime="1.094417055s" totalInitContainerRuntime="70.820196ms" isStatefulPod=true podCreationTimestamp="2026-09-07 11:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:07.561014635 +0000 UTC m=+100.035371014" watchObservedRunningTime="2026-09-07 11:02:07.56192796 +0000 UTC m=+100.036284340" Sep 07 11:02:07.583967 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.583908 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" podStartSLOduration=78.583896284 podStartE2EDuration="1m18.583896284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:07.583076243 +0000 UTC m=+100.057432622" watchObservedRunningTime="2026-09-07 11:02:07.583896284 +0000 UTC m=+100.058252660" Sep 07 11:02:07.761584 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.761542 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-cc58458f6-sd77f"] Sep 07 11:02:07.764656 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.764638 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:07.767333 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.767292 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-64co14046ndtt\"" Sep 07 11:02:07.767431 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.767373 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 07 11:02:07.767431 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.767392 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 07 11:02:07.767559 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.767472 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fqd7t\"" Sep 07 11:02:07.767612 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.767563 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 07 11:02:07.767665 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.767613 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 07 11:02:07.775156 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.775134 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-cc58458f6-sd77f"] Sep 07 11:02:07.954819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.954740 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/39c89bda-7d1b-4a28-8f38-ce22fc364f16-secret-metrics-server-client-certs\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:07.954819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.954780 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/39c89bda-7d1b-4a28-8f38-ce22fc364f16-audit-log\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:07.955019 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.954834 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/39c89bda-7d1b-4a28-8f38-ce22fc364f16-secret-metrics-server-tls\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:07.955019 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.954874 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mmtw\" (UniqueName: \"kubernetes.io/projected/39c89bda-7d1b-4a28-8f38-ce22fc364f16-kube-api-access-4mmtw\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:07.955019 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.954912 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/39c89bda-7d1b-4a28-8f38-ce22fc364f16-metrics-server-audit-profiles\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:07.955019 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.954943 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/39c89bda-7d1b-4a28-8f38-ce22fc364f16-client-ca-bundle\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:07.955214 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:07.955026 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39c89bda-7d1b-4a28-8f38-ce22fc364f16-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.056187 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.056153 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39c89bda-7d1b-4a28-8f38-ce22fc364f16-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.056187 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.056187 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/39c89bda-7d1b-4a28-8f38-ce22fc364f16-secret-metrics-server-client-certs\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.056430 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.056206 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/39c89bda-7d1b-4a28-8f38-ce22fc364f16-audit-log\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.056430 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.056251 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/39c89bda-7d1b-4a28-8f38-ce22fc364f16-secret-metrics-server-tls\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.056430 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.056293 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mmtw\" (UniqueName: \"kubernetes.io/projected/39c89bda-7d1b-4a28-8f38-ce22fc364f16-kube-api-access-4mmtw\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.056430 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.056323 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/39c89bda-7d1b-4a28-8f38-ce22fc364f16-metrics-server-audit-profiles\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.056430 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.056354 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/39c89bda-7d1b-4a28-8f38-ce22fc364f16-client-ca-bundle\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.056810 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.056782 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/39c89bda-7d1b-4a28-8f38-ce22fc364f16-audit-log\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.056972 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.056950 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39c89bda-7d1b-4a28-8f38-ce22fc364f16-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.057373 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.057353 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/39c89bda-7d1b-4a28-8f38-ce22fc364f16-metrics-server-audit-profiles\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.058990 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.058969 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/39c89bda-7d1b-4a28-8f38-ce22fc364f16-client-ca-bundle\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.059088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.059037 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/39c89bda-7d1b-4a28-8f38-ce22fc364f16-secret-metrics-server-client-certs\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.059132 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.059113 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/39c89bda-7d1b-4a28-8f38-ce22fc364f16-secret-metrics-server-tls\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.064936 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.064911 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mmtw\" (UniqueName: \"kubernetes.io/projected/39c89bda-7d1b-4a28-8f38-ce22fc364f16-kube-api-access-4mmtw\") pod \"metrics-server-cc58458f6-sd77f\" (UID: \"39c89bda-7d1b-4a28-8f38-ce22fc364f16\") " pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.074611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.074587 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:08.178512 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.178478 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd"] Sep 07 11:02:08.182542 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.182521 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" Sep 07 11:02:08.185143 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.185124 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 07 11:02:08.185252 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.185131 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-44q6s\"" Sep 07 11:02:08.194085 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.194064 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd"] Sep 07 11:02:08.236275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.236203 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dd4d7ff94-btjx6"] Sep 07 11:02:08.352983 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.352908 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-cc58458f6-sd77f"] Sep 07 11:02:08.358496 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.358465 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9593324d-c084-43ab-a4cd-c47036faff90-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-m5tkd\" (UID: \"9593324d-c084-43ab-a4cd-c47036faff90\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" Sep 07 11:02:08.358613 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.358536 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xxpj\" (UniqueName: \"kubernetes.io/projected/9593324d-c084-43ab-a4cd-c47036faff90-kube-api-access-6xxpj\") pod \"monitoring-plugin-85bbbbbf4-m5tkd\" (UID: \"9593324d-c084-43ab-a4cd-c47036faff90\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" Sep 07 11:02:08.459446 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.459403 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9593324d-c084-43ab-a4cd-c47036faff90-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-m5tkd\" (UID: \"9593324d-c084-43ab-a4cd-c47036faff90\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" Sep 07 11:02:08.459683 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.459479 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xxpj\" (UniqueName: \"kubernetes.io/projected/9593324d-c084-43ab-a4cd-c47036faff90-kube-api-access-6xxpj\") pod \"monitoring-plugin-85bbbbbf4-m5tkd\" (UID: \"9593324d-c084-43ab-a4cd-c47036faff90\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" Sep 07 11:02:08.462357 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.462324 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9593324d-c084-43ab-a4cd-c47036faff90-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-m5tkd\" (UID: \"9593324d-c084-43ab-a4cd-c47036faff90\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" Sep 07 11:02:08.471581 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.471535 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xxpj\" (UniqueName: \"kubernetes.io/projected/9593324d-c084-43ab-a4cd-c47036faff90-kube-api-access-6xxpj\") pod \"monitoring-plugin-85bbbbbf4-m5tkd\" (UID: \"9593324d-c084-43ab-a4cd-c47036faff90\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" Sep 07 11:02:08.493366 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.493306 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" Sep 07 11:02:08.642153 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:02:08.642110 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c89bda_7d1b_4a28_8f38_ce22fc364f16.slice/crio-abd2794a85bc2870e7881b8e55c7d280b9376747c69a49e1ff752ebfd6f8b6b1 WatchSource:0}: Error finding container abd2794a85bc2870e7881b8e55c7d280b9376747c69a49e1ff752ebfd6f8b6b1: Status 404 returned error can't find the container with id abd2794a85bc2870e7881b8e55c7d280b9376747c69a49e1ff752ebfd6f8b6b1 Sep 07 11:02:08.664203 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.664176 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-96b5d9885-tst5d"] Sep 07 11:02:08.667691 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.667670 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.671480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.671098 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 07 11:02:08.671480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.671179 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 07 11:02:08.671480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.671236 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8smj8\"" Sep 07 11:02:08.671480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.671108 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 07 11:02:08.671480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.671425 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 07 11:02:08.671831 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.671630 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 07 11:02:08.676021 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.676000 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 07 11:02:08.681766 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.681744 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-96b5d9885-tst5d"] Sep 07 11:02:08.762900 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.762875 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d0742af-561c-4c78-8b4b-9198b3d980fb-metrics-client-ca\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.763023 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.762906 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d0742af-561c-4c78-8b4b-9198b3d980fb-serving-certs-ca-bundle\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.763023 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.762931 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-telemeter-client-tls\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.763023 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.762945 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjqng\" (UniqueName: \"kubernetes.io/projected/5d0742af-561c-4c78-8b4b-9198b3d980fb-kube-api-access-jjqng\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.763023 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.763015 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-secret-telemeter-client\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.763232 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.763058 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d0742af-561c-4c78-8b4b-9198b3d980fb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.763232 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.763105 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-federate-client-tls\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.763232 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.763172 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.864229 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.864200 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-telemeter-client-tls\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.864339 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.864243 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjqng\" (UniqueName: \"kubernetes.io/projected/5d0742af-561c-4c78-8b4b-9198b3d980fb-kube-api-access-jjqng\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.864385 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.864345 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-secret-telemeter-client\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.864434 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.864401 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d0742af-561c-4c78-8b4b-9198b3d980fb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.864480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.864442 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-federate-client-tls\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.864480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.864463 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.864601 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.864501 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d0742af-561c-4c78-8b4b-9198b3d980fb-metrics-client-ca\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.864601 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.864521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d0742af-561c-4c78-8b4b-9198b3d980fb-serving-certs-ca-bundle\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.865306 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.865275 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d0742af-561c-4c78-8b4b-9198b3d980fb-serving-certs-ca-bundle\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.865519 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.865483 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d0742af-561c-4c78-8b4b-9198b3d980fb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.865636 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.865606 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d0742af-561c-4c78-8b4b-9198b3d980fb-metrics-client-ca\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.867166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.867138 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-telemeter-client-tls\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.867367 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.867349 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-federate-client-tls\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.867559 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.867531 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.867627 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.867607 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5d0742af-561c-4c78-8b4b-9198b3d980fb-secret-telemeter-client\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.872296 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.872277 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjqng\" (UniqueName: \"kubernetes.io/projected/5d0742af-561c-4c78-8b4b-9198b3d980fb-kube-api-access-jjqng\") pod \"telemeter-client-96b5d9885-tst5d\" (UID: \"5d0742af-561c-4c78-8b4b-9198b3d980fb\") " pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:08.912942 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.912851 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd"] Sep 07 11:02:08.915656 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:02:08.915623 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9593324d_c084_43ab_a4cd_c47036faff90.slice/crio-23afa4183682ece71681f5e0a741e396e603dea95482528fd98ffeba11a07a5a WatchSource:0}: Error finding container 23afa4183682ece71681f5e0a741e396e603dea95482528fd98ffeba11a07a5a: Status 404 returned error can't find the container with id 23afa4183682ece71681f5e0a741e396e603dea95482528fd98ffeba11a07a5a Sep 07 11:02:08.982911 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:08.982877 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" Sep 07 11:02:09.276043 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:09.276014 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-96b5d9885-tst5d"] Sep 07 11:02:09.279058 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:02:09.279029 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0742af_561c_4c78_8b4b_9198b3d980fb.slice/crio-7ce4556a7de0a71e16dcc96292befc31554e339943a85ee17b7dd0b35f707d34 WatchSource:0}: Error finding container 7ce4556a7de0a71e16dcc96292befc31554e339943a85ee17b7dd0b35f707d34: Status 404 returned error can't find the container with id 7ce4556a7de0a71e16dcc96292befc31554e339943a85ee17b7dd0b35f707d34 Sep 07 11:02:09.552250 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:09.552148 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" event={"ID":"5d0742af-561c-4c78-8b4b-9198b3d980fb","Type":"ContainerStarted","Data":"7ce4556a7de0a71e16dcc96292befc31554e339943a85ee17b7dd0b35f707d34"} Sep 07 11:02:09.554255 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:09.554206 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-49lhf" event={"ID":"cd09d09c-a0d7-4104-8d79-746a7d0b8030","Type":"ContainerStarted","Data":"6aee73d2277dbf6f2a2d23681738c98a466f47630516158cebe1919a158950d4"} Sep 07 11:02:09.554404 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:09.554275 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-49lhf" event={"ID":"cd09d09c-a0d7-4104-8d79-746a7d0b8030","Type":"ContainerStarted","Data":"aaa5fe2513de2aa414d16b07a3b5a3f5a0d21dfeed6a941eaad5ff7a8c1d6646"} Sep 07 11:02:09.554404 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:09.554306 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-49lhf" Sep 07 11:02:09.555754 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:09.555726 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" event={"ID":"39c89bda-7d1b-4a28-8f38-ce22fc364f16","Type":"ContainerStarted","Data":"abd2794a85bc2870e7881b8e55c7d280b9376747c69a49e1ff752ebfd6f8b6b1"} Sep 07 11:02:09.557297 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:09.557265 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" event={"ID":"9593324d-c084-43ab-a4cd-c47036faff90","Type":"ContainerStarted","Data":"23afa4183682ece71681f5e0a741e396e603dea95482528fd98ffeba11a07a5a"} Sep 07 11:02:09.572322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:09.572220 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-49lhf" podStartSLOduration=65.819103333 podStartE2EDuration="1m7.57218096s" totalImagesPullingTime="1.753077627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:09.570300942 +0000 UTC m=+102.044657320" watchObservedRunningTime="2026-09-07 11:02:09.57218096 +0000 UTC m=+102.046537334" Sep 07 11:02:10.412972 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:10.412939 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qz5nc" Sep 07 11:02:12.570690 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:12.570652 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" event={"ID":"9593324d-c084-43ab-a4cd-c47036faff90","Type":"ContainerStarted","Data":"4e60acb3662e10bdbf44806f9497a57f94cc573aeae6080e604b238f388273ae"} Sep 07 11:02:12.571115 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:12.570881 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" Sep 07 11:02:12.572360 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:12.572333 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" event={"ID":"5d0742af-561c-4c78-8b4b-9198b3d980fb","Type":"ContainerStarted","Data":"c6f99b0af4ae2f6d96e9445cd850173d6287b2379cd539d42f99e526603aeef4"} Sep 07 11:02:12.574053 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:12.574027 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" event={"ID":"39c89bda-7d1b-4a28-8f38-ce22fc364f16","Type":"ContainerStarted","Data":"88bf113d4ca8dc4947dd4200c4095a137a0208095594afccb916f627841f9982"} Sep 07 11:02:12.575900 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:12.575877 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" Sep 07 11:02:12.585881 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:12.585844 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m5tkd" podStartSLOduration=1.451689028 podStartE2EDuration="4.58582934s" totalImagesPullingTime="3.134140312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:12.584467384 +0000 UTC m=+105.058823765" watchObservedRunningTime="2026-09-07 11:02:12.58582934 +0000 UTC m=+105.060185718" Sep 07 11:02:12.615007 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:12.614963 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" podStartSLOduration=2.233300011 podStartE2EDuration="5.614947549s" totalImagesPullingTime="3.381647538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:12.613842555 +0000 UTC m=+105.088198937" watchObservedRunningTime="2026-09-07 11:02:12.614947549 +0000 UTC m=+105.089303926" Sep 07 11:02:13.579458 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:13.579368 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" event={"ID":"5d0742af-561c-4c78-8b4b-9198b3d980fb","Type":"ContainerStarted","Data":"261d2ca70a53518248b867f9f883a046406ad1e857991eb3da2068dcc8a8e729"} Sep 07 11:02:13.579838 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:13.579459 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" event={"ID":"5d0742af-561c-4c78-8b4b-9198b3d980fb","Type":"ContainerStarted","Data":"aaca00fb6d4c3feb1db6270a96c894490ffea3d18fe17c3358a058d08916d0c0"} Sep 07 11:02:13.600754 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:13.600697 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-96b5d9885-tst5d" podStartSLOduration=1.784841565 podStartE2EDuration="5.60067943s" totalImagesPullingTime="3.815837865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:02:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:13.599280095 +0000 UTC m=+106.073636496" watchObservedRunningTime="2026-09-07 11:02:13.60067943 +0000 UTC m=+106.075035810" Sep 07 11:02:19.563688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:19.563658 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-49lhf" Sep 07 11:02:28.075442 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:28.075399 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:28.075926 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:28.075492 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:28.636161 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:28.636130 2571 generic.go:356] "Generic (PLEG): container finished" podID="ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d" containerID="47047dc7910e50b64ad6768ce62b5f3ef52a71728ba5c9f993d497ee972f3f04" exitCode=0 Sep 07 11:02:28.636335 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:28.636205 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" event={"ID":"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d","Type":"ContainerDied","Data":"47047dc7910e50b64ad6768ce62b5f3ef52a71728ba5c9f993d497ee972f3f04"} Sep 07 11:02:28.636585 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:28.636539 2571 scope.go:117] "RemoveContainer" containerID="47047dc7910e50b64ad6768ce62b5f3ef52a71728ba5c9f993d497ee972f3f04" Sep 07 11:02:29.563322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:29.563291 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:02:29.641655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:29.641625 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lm4tc" event={"ID":"ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d","Type":"ContainerStarted","Data":"2656f422ebb4b0d8142dcd6f4f995049304665fb717e3cb40d8d2f36c22b94cb"} Sep 07 11:02:34.583725 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.583683 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" podUID="5a201c33-d8a9-4792-9add-57056228abf6" containerName="registry" containerID="cri-o://feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5" gracePeriod=30 Sep 07 11:02:34.816517 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.816489 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:02:34.876879 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.876804 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") pod \"5a201c33-d8a9-4792-9add-57056228abf6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " Sep 07 11:02:34.876879 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.876844 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-image-registry-private-configuration\") pod \"5a201c33-d8a9-4792-9add-57056228abf6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " Sep 07 11:02:34.877091 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.876886 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrd4x\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-kube-api-access-rrd4x\") pod \"5a201c33-d8a9-4792-9add-57056228abf6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " Sep 07 11:02:34.877091 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.876926 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a201c33-d8a9-4792-9add-57056228abf6-ca-trust-extracted\") pod \"5a201c33-d8a9-4792-9add-57056228abf6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " Sep 07 11:02:34.877091 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.876949 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-installation-pull-secrets\") pod \"5a201c33-d8a9-4792-9add-57056228abf6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " Sep 07 11:02:34.877091 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.876968 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-bound-sa-token\") pod \"5a201c33-d8a9-4792-9add-57056228abf6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " Sep 07 11:02:34.877091 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.877016 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-trusted-ca\") pod \"5a201c33-d8a9-4792-9add-57056228abf6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " Sep 07 11:02:34.877091 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.877041 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-registry-certificates\") pod \"5a201c33-d8a9-4792-9add-57056228abf6\" (UID: \"5a201c33-d8a9-4792-9add-57056228abf6\") " Sep 07 11:02:34.877578 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.877528 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5a201c33-d8a9-4792-9add-57056228abf6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 11:02:34.877753 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.877542 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5a201c33-d8a9-4792-9add-57056228abf6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 11:02:34.879236 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.879211 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5a201c33-d8a9-4792-9add-57056228abf6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:02:34.879537 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.879504 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5a201c33-d8a9-4792-9add-57056228abf6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:02:34.879699 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.879572 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-kube-api-access-rrd4x" (OuterVolumeSpecName: "kube-api-access-rrd4x") pod "5a201c33-d8a9-4792-9add-57056228abf6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6"). InnerVolumeSpecName "kube-api-access-rrd4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:02:34.879699 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.879615 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5a201c33-d8a9-4792-9add-57056228abf6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:02:34.879699 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.879661 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5a201c33-d8a9-4792-9add-57056228abf6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:02:34.887624 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.887599 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a201c33-d8a9-4792-9add-57056228abf6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5a201c33-d8a9-4792-9add-57056228abf6" (UID: "5a201c33-d8a9-4792-9add-57056228abf6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:02:34.978115 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.978083 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrd4x\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-kube-api-access-rrd4x\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:02:34.978115 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.978109 2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a201c33-d8a9-4792-9add-57056228abf6-ca-trust-extracted\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:02:34.978115 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.978118 2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-installation-pull-secrets\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:02:34.978292 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.978128 2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-bound-sa-token\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:02:34.978292 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.978138 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-trusted-ca\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:02:34.978292 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.978145 2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a201c33-d8a9-4792-9add-57056228abf6-registry-certificates\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:02:34.978292 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.978154 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a201c33-d8a9-4792-9add-57056228abf6-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:02:34.978292 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:34.978163 2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a201c33-d8a9-4792-9add-57056228abf6-image-registry-private-configuration\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:02:35.664964 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:35.664926 2571 generic.go:356] "Generic (PLEG): container finished" podID="5a201c33-d8a9-4792-9add-57056228abf6" containerID="feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5" exitCode=0 Sep 07 11:02:35.665380 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:35.664972 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" Sep 07 11:02:35.665380 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:35.665006 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" event={"ID":"5a201c33-d8a9-4792-9add-57056228abf6","Type":"ContainerDied","Data":"feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5"} Sep 07 11:02:35.665380 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:35.665054 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd4d7ff94-btjx6" event={"ID":"5a201c33-d8a9-4792-9add-57056228abf6","Type":"ContainerDied","Data":"8991d03184813c4ef24f9bd8a65a6687949c5d4d02b8645a13afd9a89afe1a0f"} Sep 07 11:02:35.665380 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:35.665073 2571 scope.go:117] "RemoveContainer" containerID="feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5" Sep 07 11:02:35.673506 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:35.673490 2571 scope.go:117] "RemoveContainer" containerID="feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5" Sep 07 11:02:35.673785 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:02:35.673768 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5\": container with ID starting with feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5 not found: ID does not exist" containerID="feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5" Sep 07 11:02:35.673845 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:35.673793 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5"} err="failed to get container status \"feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5\": rpc error: code = NotFound desc = could not find container \"feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5\": container with ID starting with feb4f72ee45853e6066f77be243e7919281d36e5e071fc4e5a4d2f9b1e67acf5 not found: ID does not exist" Sep 07 11:02:35.686023 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:35.686001 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dd4d7ff94-btjx6"] Sep 07 11:02:35.691255 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:35.691236 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-dd4d7ff94-btjx6"] Sep 07 11:02:36.123327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:36.123298 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a201c33-d8a9-4792-9add-57056228abf6" path="/var/lib/kubelet/pods/5a201c33-d8a9-4792-9add-57056228abf6/volumes" Sep 07 11:02:37.801542 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:37.801509 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:02:37.803693 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:37.803673 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0-metrics-certs\") pod \"network-metrics-daemon-6jc59\" (UID: \"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0\") " pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:02:38.047409 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:38.047382 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m22ww\"" Sep 07 11:02:38.055527 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:38.055476 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jc59" Sep 07 11:02:38.322056 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:38.321985 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jc59"] Sep 07 11:02:38.324817 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:02:38.324793 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3d1c4d_4f6c_40a3_bd80_ee89c163edf0.slice/crio-7eda1f6d816228545e45fc72e3ac4b58925dae0af306b29efd5252b96631d600 WatchSource:0}: Error finding container 7eda1f6d816228545e45fc72e3ac4b58925dae0af306b29efd5252b96631d600: Status 404 returned error can't find the container with id 7eda1f6d816228545e45fc72e3ac4b58925dae0af306b29efd5252b96631d600 Sep 07 11:02:38.677508 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:38.677439 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jc59" event={"ID":"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0","Type":"ContainerStarted","Data":"7eda1f6d816228545e45fc72e3ac4b58925dae0af306b29efd5252b96631d600"} Sep 07 11:02:39.682427 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:39.682359 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jc59" event={"ID":"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0","Type":"ContainerStarted","Data":"dfdeec8c6b2dce40a250103bb376ee19a2c27441c8fe72538913b98bade5630f"} Sep 07 11:02:39.682427 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:39.682406 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jc59" event={"ID":"cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0","Type":"ContainerStarted","Data":"d5e9ba7687a403cdb65c29385b51d7bcc42bb67886bcc7e78710c452570aae4c"} Sep 07 11:02:39.701244 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:39.701199 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6jc59" podStartSLOduration=130.655837519 podStartE2EDuration="2m11.701183992s" totalImagesPullingTime="1.045346473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:39.700495076 +0000 UTC m=+132.174851454" watchObservedRunningTime="2026-09-07 11:02:39.701183992 +0000 UTC m=+132.175540765" Sep 07 11:02:46.470941 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:46.470912 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sjcrj_e23c9000-ee54-41ff-8409-5da16eed5f60/serve-healthcheck-canary/0.log" Sep 07 11:02:48.080624 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:48.080593 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:48.084216 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:48.084192 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-cc58458f6-sd77f" Sep 07 11:02:54.740264 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:54.740227 2571 generic.go:356] "Generic (PLEG): container finished" podID="f55c689e-68fb-4fbb-914f-dae8863f36cc" containerID="5be70cc8e18087858d2e43a6d63b0efa156ae084c288913d159eb319d44c1d76" exitCode=0 Sep 07 11:02:54.740682 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:54.740298 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" event={"ID":"f55c689e-68fb-4fbb-914f-dae8863f36cc","Type":"ContainerDied","Data":"5be70cc8e18087858d2e43a6d63b0efa156ae084c288913d159eb319d44c1d76"} Sep 07 11:02:54.740682 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:54.740639 2571 scope.go:117] "RemoveContainer" containerID="5be70cc8e18087858d2e43a6d63b0efa156ae084c288913d159eb319d44c1d76" Sep 07 11:02:55.745466 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:02:55.745427 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kv4fh" event={"ID":"f55c689e-68fb-4fbb-914f-dae8863f36cc","Type":"ContainerStarted","Data":"bcd987c90908df72e707a3f437c7e670d4e6a63a9e69553da4e33fdff6e3f6bb"} Sep 07 11:05:27.945605 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:05:27.945576 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:05:27.952381 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:05:27.952354 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:05:27.957394 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:05:27.957374 2571 kubelet.go:1648] "Image garbage collection succeeded" Sep 07 11:06:16.509114 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.509083 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5"] Sep 07 11:06:16.509535 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.509372 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a201c33-d8a9-4792-9add-57056228abf6" containerName="registry" Sep 07 11:06:16.509535 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.509382 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a201c33-d8a9-4792-9add-57056228abf6" containerName="registry" Sep 07 11:06:16.509535 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.509428 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a201c33-d8a9-4792-9add-57056228abf6" containerName="registry" Sep 07 11:06:16.512127 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.512108 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" Sep 07 11:06:16.514475 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.514456 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 07 11:06:16.514570 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.514495 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 07 11:06:16.515643 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.515625 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-d7wcm\"" Sep 07 11:06:16.529711 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.529688 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5"] Sep 07 11:06:16.692784 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.692753 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wlz\" (UniqueName: \"kubernetes.io/projected/085001ba-6664-4434-be0b-fee420861e51-kube-api-access-p5wlz\") pod \"feast-operator-controller-manager-7fd64c9d98-cg5p5\" (UID: \"085001ba-6664-4434-be0b-fee420861e51\") " pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" Sep 07 11:06:16.794002 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.793973 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wlz\" (UniqueName: \"kubernetes.io/projected/085001ba-6664-4434-be0b-fee420861e51-kube-api-access-p5wlz\") pod \"feast-operator-controller-manager-7fd64c9d98-cg5p5\" (UID: \"085001ba-6664-4434-be0b-fee420861e51\") " pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" Sep 07 11:06:16.802375 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.802351 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wlz\" (UniqueName: \"kubernetes.io/projected/085001ba-6664-4434-be0b-fee420861e51-kube-api-access-p5wlz\") pod \"feast-operator-controller-manager-7fd64c9d98-cg5p5\" (UID: \"085001ba-6664-4434-be0b-fee420861e51\") " pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" Sep 07 11:06:16.822276 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:16.822252 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" Sep 07 11:06:17.089132 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:17.089089 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5"] Sep 07 11:06:17.091368 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:06:17.091334 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085001ba_6664_4434_be0b_fee420861e51.slice/crio-70769682260ad09f463be47b8a7d2271f4204f3adcb649add528802a5fbed4c5 WatchSource:0}: Error finding container 70769682260ad09f463be47b8a7d2271f4204f3adcb649add528802a5fbed4c5: Status 404 returned error can't find the container with id 70769682260ad09f463be47b8a7d2271f4204f3adcb649add528802a5fbed4c5 Sep 07 11:06:17.093328 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:17.093310 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 11:06:17.469286 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:17.469208 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" event={"ID":"085001ba-6664-4434-be0b-fee420861e51","Type":"ContainerStarted","Data":"70769682260ad09f463be47b8a7d2271f4204f3adcb649add528802a5fbed4c5"} Sep 07 11:06:19.481536 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:19.481496 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" event={"ID":"085001ba-6664-4434-be0b-fee420861e51","Type":"ContainerStarted","Data":"574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee"} Sep 07 11:06:19.482354 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:19.482330 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" Sep 07 11:06:19.499191 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:19.499140 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" podStartSLOduration=1.27223567 podStartE2EDuration="3.499126586s" totalImagesPullingTime="2.226890916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:06:19.498778197 +0000 UTC m=+351.973134580" watchObservedRunningTime="2026-09-07 11:06:19.499126586 +0000 UTC m=+351.973483009" Sep 07 11:06:31.492065 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:31.491977 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" Sep 07 11:06:56.438367 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.438332 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5"] Sep 07 11:06:56.442439 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.442417 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.444867 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.444842 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 07 11:06:56.444980 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.444851 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 07 11:06:56.445063 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.445042 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 07 11:06:56.445128 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.445075 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 07 11:06:56.445186 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.445174 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 07 11:06:56.445281 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.445261 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 07 11:06:56.445390 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.445297 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 07 11:06:56.445390 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.445380 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-nqll6\"" Sep 07 11:06:56.451744 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.451724 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5"] Sep 07 11:06:56.494378 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.494349 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-ui-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.494520 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.494391 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-registry-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.494520 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.494416 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5pt\" (UniqueName: \"kubernetes.io/projected/42e9614b-0672-41cc-8658-48934d214f4b-kube-api-access-kg5pt\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.494520 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.494493 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-online-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.494711 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.494534 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/42e9614b-0672-41cc-8658-48934d214f4b-feast-data\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.494711 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.494632 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-offline-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.595334 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.595302 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-online-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.595451 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.595354 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/42e9614b-0672-41cc-8658-48934d214f4b-feast-data\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.595451 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.595400 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-offline-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.595451 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.595440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-ui-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.595611 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:06:56.595452 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 07 11:06:56.595611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.595469 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-registry-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.595611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.595496 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5pt\" (UniqueName: \"kubernetes.io/projected/42e9614b-0672-41cc-8658-48934d214f4b-kube-api-access-kg5pt\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.595611 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:06:56.595516 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 07 11:06:56.595611 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:06:56.595526 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-online-tls podName:42e9614b-0672-41cc-8658-48934d214f4b nodeName:}" failed. No retries permitted until 2026-09-07 11:06:57.095503284 +0000 UTC m=+389.569859655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-online-tls") pod "feast-simple-feast-setup-6bc7846c6c-z2cv5" (UID: "42e9614b-0672-41cc-8658-48934d214f4b") : secret "feast-simple-feast-setup-online-tls" not found Sep 07 11:06:56.595611 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:06:56.595593 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-offline-tls podName:42e9614b-0672-41cc-8658-48934d214f4b nodeName:}" failed. No retries permitted until 2026-09-07 11:06:57.09557676 +0000 UTC m=+389.569933131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-offline-tls") pod "feast-simple-feast-setup-6bc7846c6c-z2cv5" (UID: "42e9614b-0672-41cc-8658-48934d214f4b") : secret "feast-simple-feast-setup-offline-tls" not found Sep 07 11:06:56.595911 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.595824 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/42e9614b-0672-41cc-8658-48934d214f4b-feast-data\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.597926 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.597894 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-registry-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.598027 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.597964 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-ui-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:56.603659 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:56.603634 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5pt\" (UniqueName: \"kubernetes.io/projected/42e9614b-0672-41cc-8658-48934d214f4b-kube-api-access-kg5pt\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:57.100164 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:57.100130 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-online-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:57.100334 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:57.100195 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-offline-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:57.102659 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:57.102631 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-online-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:57.102777 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:57.102660 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-offline-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-z2cv5\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:57.352603 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:57.352503 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:06:57.633486 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:57.633461 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5"] Sep 07 11:06:57.637293 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:06:57.636162 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42e9614b_0672_41cc_8658_48934d214f4b.slice/crio-66cff2d32eeb133fa86643f569b23528a64ff8449a2e60a537ec0ea32c0ea9b0 WatchSource:0}: Error finding container 66cff2d32eeb133fa86643f569b23528a64ff8449a2e60a537ec0ea32c0ea9b0: Status 404 returned error can't find the container with id 66cff2d32eeb133fa86643f569b23528a64ff8449a2e60a537ec0ea32c0ea9b0 Sep 07 11:06:58.625569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:06:58.625519 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerStarted","Data":"66cff2d32eeb133fa86643f569b23528a64ff8449a2e60a537ec0ea32c0ea9b0"} Sep 07 11:07:24.741516 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:24.741477 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerStarted","Data":"26f6d2c53d23b622b40c4e37b28e36e6848ccee05b535a15017337ffb17bbe6c"} Sep 07 11:07:29.764542 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:29.764504 2571 generic.go:356] "Generic (PLEG): container finished" podID="42e9614b-0672-41cc-8658-48934d214f4b" containerID="26f6d2c53d23b622b40c4e37b28e36e6848ccee05b535a15017337ffb17bbe6c" exitCode=0 Sep 07 11:07:29.765059 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:29.764582 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerDied","Data":"26f6d2c53d23b622b40c4e37b28e36e6848ccee05b535a15017337ffb17bbe6c"} Sep 07 11:07:30.770251 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:30.770211 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerStarted","Data":"41778fb220c03f34d756ed91e202ded8e72d18ed67398ebee03c416002f60890"} Sep 07 11:07:35.791964 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:35.791926 2571 generic.go:356] "Generic (PLEG): container finished" podID="42e9614b-0672-41cc-8658-48934d214f4b" containerID="41778fb220c03f34d756ed91e202ded8e72d18ed67398ebee03c416002f60890" exitCode=0 Sep 07 11:07:35.792340 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:35.791970 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerDied","Data":"41778fb220c03f34d756ed91e202ded8e72d18ed67398ebee03c416002f60890"} Sep 07 11:07:35.795817 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:35.795791 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 07 11:07:36.802663 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:36.802620 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerStarted","Data":"94089da790aadcc1da720b5c2de9928089a88cb579e49bb5ab1f69ff7dc03a12"} Sep 07 11:07:36.803255 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:36.802687 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerStarted","Data":"9fc728931cda994cc57c9c17ca0b4c61fc1d13a0fba0ecf22e81e0b574e821f0"} Sep 07 11:07:36.803255 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:36.802709 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerStarted","Data":"784b13551d36aeba410f235c5d12f7f2886687b5aa38f6accf1d44fed1b183fe"} Sep 07 11:07:36.803255 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:36.802734 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerStarted","Data":"03d1b2a81c328a7dd7d619852e51e76a3acbbcd2affcb9609e1e3bff5723c485"} Sep 07 11:07:36.830123 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:36.829995 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podStartSLOduration=13.877422379 podStartE2EDuration="40.829936028s" totalImagesPullingTime="26.952513649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:07:36.827831235 +0000 UTC m=+429.302187615" watchObservedRunningTime="2026-09-07 11:07:36.829936028 +0000 UTC m=+429.304292413" Sep 07 11:07:37.352953 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:37.352908 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:37.353250 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:37.353221 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:37.353379 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:37.353256 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:37.353379 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:37.353277 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:39.353778 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:39.353737 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:39.354365 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:39.354335 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:39.354494 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:39.354371 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:39.354494 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:39.354392 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:39.356123 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:39.356027 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.22:8443: connect: connection refused" Sep 07 11:07:39.356123 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:39.356073 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.22:6571: connect: connection refused" Sep 07 11:07:39.356299 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:39.356074 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.22:6567/health\": dial tcp 10.132.0.22:6567: connect: connection refused" Sep 07 11:07:39.356299 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:39.356083 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.22:8816: connect: connection refused" Sep 07 11:07:42.353144 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:42.353087 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.22:8443: connect: connection refused" Sep 07 11:07:42.353671 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:42.353096 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.22:6571: connect: connection refused" Sep 07 11:07:42.353671 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:42.353098 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.22:6567/health\": dial tcp 10.132.0.22:6567: connect: connection refused" Sep 07 11:07:42.353910 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:42.353874 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.22:8816: connect: connection refused" Sep 07 11:07:45.353827 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:45.353794 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:45.354312 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:45.353987 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:45.354312 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:45.354213 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:45.354886 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:45.354862 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:45.355142 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:45.355120 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:45.355702 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:45.355683 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:45.359565 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:45.359535 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:07:45.852626 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:07:45.852593 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:08:48.784129 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:48.784078 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5"] Sep 07 11:08:48.784681 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:48.784379 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="registry" containerID="cri-o://03d1b2a81c328a7dd7d619852e51e76a3acbbcd2affcb9609e1e3bff5723c485" gracePeriod=30 Sep 07 11:08:48.784681 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:48.784431 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="online" containerID="cri-o://784b13551d36aeba410f235c5d12f7f2886687b5aa38f6accf1d44fed1b183fe" gracePeriod=30 Sep 07 11:08:48.784681 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:48.784448 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="ui" containerID="cri-o://94089da790aadcc1da720b5c2de9928089a88cb579e49bb5ab1f69ff7dc03a12" gracePeriod=30 Sep 07 11:08:48.784885 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:48.784463 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="offline" containerID="cri-o://9fc728931cda994cc57c9c17ca0b4c61fc1d13a0fba0ecf22e81e0b574e821f0" gracePeriod=30 Sep 07 11:08:49.227636 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.227543 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s"] Sep 07 11:08:49.231817 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.231792 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.234252 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.234225 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-95ns4\"" Sep 07 11:08:49.243877 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.243840 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s"] Sep 07 11:08:49.309961 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.309927 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-online-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.310156 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.309974 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqjqb\" (UniqueName: \"kubernetes.io/projected/87641106-863f-412e-b8e2-cc370ec438ee-kube-api-access-kqjqb\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.310156 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.309994 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-registry-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.310156 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.310083 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-offline-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.310156 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.310117 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-ui-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.310371 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.310162 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/87641106-863f-412e-b8e2-cc370ec438ee-feast-data\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.410781 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.410745 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-online-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.410979 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.410790 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqjqb\" (UniqueName: \"kubernetes.io/projected/87641106-863f-412e-b8e2-cc370ec438ee-kube-api-access-kqjqb\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.410979 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.410809 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-registry-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.410979 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:08:49.410914 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 07 11:08:49.410979 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.410938 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-offline-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.411184 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:08:49.410990 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-online-tls podName:87641106-863f-412e-b8e2-cc370ec438ee nodeName:}" failed. No retries permitted until 2026-09-07 11:08:49.910970652 +0000 UTC m=+502.385327037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-online-tls") pod "feast-simple-feast-setup-6bc7846c6c-m722s" (UID: "87641106-863f-412e-b8e2-cc370ec438ee") : secret "feast-simple-feast-setup-online-tls" not found Sep 07 11:08:49.411184 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:08:49.411009 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:08:49.411184 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:08:49.411059 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-registry-tls podName:87641106-863f-412e-b8e2-cc370ec438ee nodeName:}" failed. No retries permitted until 2026-09-07 11:08:49.911046529 +0000 UTC m=+502.385402891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-registry-tls") pod "feast-simple-feast-setup-6bc7846c6c-m722s" (UID: "87641106-863f-412e-b8e2-cc370ec438ee") : secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:08:49.411184 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.411075 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-ui-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.411184 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.411116 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/87641106-863f-412e-b8e2-cc370ec438ee-feast-data\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.411389 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:08:49.411244 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 07 11:08:49.411389 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:08:49.411295 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-ui-tls podName:87641106-863f-412e-b8e2-cc370ec438ee nodeName:}" failed. No retries permitted until 2026-09-07 11:08:49.911280974 +0000 UTC m=+502.385637346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-ui-tls") pod "feast-simple-feast-setup-6bc7846c6c-m722s" (UID: "87641106-863f-412e-b8e2-cc370ec438ee") : secret "feast-simple-feast-setup-ui-tls" not found Sep 07 11:08:49.411542 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.411523 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/87641106-863f-412e-b8e2-cc370ec438ee-feast-data\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.413529 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.413500 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-offline-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.420704 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.420683 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqjqb\" (UniqueName: \"kubernetes.io/projected/87641106-863f-412e-b8e2-cc370ec438ee-kube-api-access-kqjqb\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.916249 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.916194 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-ui-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.916765 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.916287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-online-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.916765 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.916315 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-registry-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.918782 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.918753 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-registry-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.918982 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.918958 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-ui-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:49.919026 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:49.918958 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-online-tls\") pod \"feast-simple-feast-setup-6bc7846c6c-m722s\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:50.095650 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:50.095613 2571 generic.go:356] "Generic (PLEG): container finished" podID="42e9614b-0672-41cc-8658-48934d214f4b" containerID="94089da790aadcc1da720b5c2de9928089a88cb579e49bb5ab1f69ff7dc03a12" exitCode=0 Sep 07 11:08:50.095650 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:50.095638 2571 generic.go:356] "Generic (PLEG): container finished" podID="42e9614b-0672-41cc-8658-48934d214f4b" containerID="784b13551d36aeba410f235c5d12f7f2886687b5aa38f6accf1d44fed1b183fe" exitCode=0 Sep 07 11:08:50.095843 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:50.095664 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerDied","Data":"94089da790aadcc1da720b5c2de9928089a88cb579e49bb5ab1f69ff7dc03a12"} Sep 07 11:08:50.095843 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:50.095694 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerDied","Data":"784b13551d36aeba410f235c5d12f7f2886687b5aa38f6accf1d44fed1b183fe"} Sep 07 11:08:50.141089 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:50.141059 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:08:50.417591 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:50.417562 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s"] Sep 07 11:08:50.418801 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:08:50.418773 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87641106_863f_412e_b8e2_cc370ec438ee.slice/crio-b844805cca40260f288df613c8ac88df2318058e77af4040f39c132edc510b8f WatchSource:0}: Error finding container b844805cca40260f288df613c8ac88df2318058e77af4040f39c132edc510b8f: Status 404 returned error can't find the container with id b844805cca40260f288df613c8ac88df2318058e77af4040f39c132edc510b8f Sep 07 11:08:51.100993 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:51.100959 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerStarted","Data":"241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681"} Sep 07 11:08:51.101361 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:51.101004 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerStarted","Data":"b844805cca40260f288df613c8ac88df2318058e77af4040f39c132edc510b8f"} Sep 07 11:08:55.355197 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:55.355146 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.22:8443: connect: connection refused" Sep 07 11:08:55.849862 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:55.849811 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.22:6567/health\": dial tcp 10.132.0.22:6567: connect: connection refused" Sep 07 11:08:56.123236 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:56.123151 2571 generic.go:356] "Generic (PLEG): container finished" podID="87641106-863f-412e-b8e2-cc370ec438ee" containerID="241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681" exitCode=0 Sep 07 11:08:56.123236 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:56.123218 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerDied","Data":"241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681"} Sep 07 11:08:57.128906 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:08:57.128871 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerStarted","Data":"cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1"} Sep 07 11:09:02.150636 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:02.150531 2571 generic.go:356] "Generic (PLEG): container finished" podID="87641106-863f-412e-b8e2-cc370ec438ee" containerID="cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1" exitCode=0 Sep 07 11:09:02.150636 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:02.150582 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerDied","Data":"cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1"} Sep 07 11:09:03.161034 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:03.160990 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerStarted","Data":"66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1"} Sep 07 11:09:03.161515 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:03.161059 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerStarted","Data":"405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d"} Sep 07 11:09:03.161515 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:03.161082 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerStarted","Data":"7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e"} Sep 07 11:09:03.161515 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:03.161102 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerStarted","Data":"727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125"} Sep 07 11:09:03.184414 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:03.184353 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podStartSLOduration=14.184333559 podStartE2EDuration="14.184333559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:09:03.182383359 +0000 UTC m=+515.656739742" watchObservedRunningTime="2026-09-07 11:09:03.184333559 +0000 UTC m=+515.658689939" Sep 07 11:09:05.141996 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:05.141953 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:05.142607 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:05.142576 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:05.142752 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:05.142614 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:05.142752 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:05.142633 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:05.144442 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:05.144401 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.23:8816: connect: connection refused" Sep 07 11:09:05.144541 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:05.144429 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="online" probeResult="failure" output="Get \"https://10.132.0.23:6567/health\": dial tcp 10.132.0.23:6567: connect: connection refused" Sep 07 11:09:05.144541 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:05.144472 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.23:8443: connect: connection refused" Sep 07 11:09:05.144541 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:05.144437 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.23:6571: connect: connection refused" Sep 07 11:09:05.355758 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:05.355596 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.22:8443: connect: connection refused" Sep 07 11:09:05.850356 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:05.850301 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.22:6567/health\": dial tcp 10.132.0.22:6567: connect: connection refused" Sep 07 11:09:08.141999 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:08.141855 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="online" probeResult="failure" output="Get \"https://10.132.0.23:6567/health\": dial tcp 10.132.0.23:6567: connect: connection refused" Sep 07 11:09:08.141999 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:08.141855 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.23:8816: connect: connection refused" Sep 07 11:09:08.141999 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:08.141872 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.23:6571: connect: connection refused" Sep 07 11:09:08.142600 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:08.142311 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.23:8443: connect: connection refused" Sep 07 11:09:10.141774 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:10.141741 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:10.142148 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:10.141793 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:10.142148 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:10.141809 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:10.142148 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:10.141820 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:11.142879 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:11.142844 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:11.143317 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:11.143027 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:11.143317 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:11.143156 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:11.143801 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:11.143776 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:11.143884 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:11.143837 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:11.144022 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:11.144004 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:11.148353 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:11.148337 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:11.212529 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:11.212499 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:09:15.354924 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:15.354881 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.22:8443: connect: connection refused" Sep 07 11:09:15.355388 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:15.355086 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:09:15.849663 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:15.849626 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.22:6567/health\": dial tcp 10.132.0.22:6567: connect: connection refused" Sep 07 11:09:15.849815 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:15.849743 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:09:19.239840 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.239743 2571 generic.go:356] "Generic (PLEG): container finished" podID="42e9614b-0672-41cc-8658-48934d214f4b" containerID="9fc728931cda994cc57c9c17ca0b4c61fc1d13a0fba0ecf22e81e0b574e821f0" exitCode=137 Sep 07 11:09:19.239840 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.239771 2571 generic.go:356] "Generic (PLEG): container finished" podID="42e9614b-0672-41cc-8658-48934d214f4b" containerID="03d1b2a81c328a7dd7d619852e51e76a3acbbcd2affcb9609e1e3bff5723c485" exitCode=137 Sep 07 11:09:19.239840 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.239818 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerDied","Data":"9fc728931cda994cc57c9c17ca0b4c61fc1d13a0fba0ecf22e81e0b574e821f0"} Sep 07 11:09:19.240249 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.239870 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerDied","Data":"03d1b2a81c328a7dd7d619852e51e76a3acbbcd2affcb9609e1e3bff5723c485"} Sep 07 11:09:19.427405 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.427382 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:09:19.565695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.565665 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-registry-tls\") pod \"42e9614b-0672-41cc-8658-48934d214f4b\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " Sep 07 11:09:19.565892 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.565704 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg5pt\" (UniqueName: \"kubernetes.io/projected/42e9614b-0672-41cc-8658-48934d214f4b-kube-api-access-kg5pt\") pod \"42e9614b-0672-41cc-8658-48934d214f4b\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " Sep 07 11:09:19.565892 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.565723 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-offline-tls\") pod \"42e9614b-0672-41cc-8658-48934d214f4b\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " Sep 07 11:09:19.565892 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.565744 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-online-tls\") pod \"42e9614b-0672-41cc-8658-48934d214f4b\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " Sep 07 11:09:19.565892 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.565778 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-ui-tls\") pod \"42e9614b-0672-41cc-8658-48934d214f4b\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " Sep 07 11:09:19.565892 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.565800 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/42e9614b-0672-41cc-8658-48934d214f4b-feast-data\") pod \"42e9614b-0672-41cc-8658-48934d214f4b\" (UID: \"42e9614b-0672-41cc-8658-48934d214f4b\") " Sep 07 11:09:19.566470 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.566439 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e9614b-0672-41cc-8658-48934d214f4b-feast-data" (OuterVolumeSpecName: "feast-data") pod "42e9614b-0672-41cc-8658-48934d214f4b" (UID: "42e9614b-0672-41cc-8658-48934d214f4b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:09:19.568156 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.568129 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "42e9614b-0672-41cc-8658-48934d214f4b" (UID: "42e9614b-0672-41cc-8658-48934d214f4b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:09:19.568250 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.568204 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "42e9614b-0672-41cc-8658-48934d214f4b" (UID: "42e9614b-0672-41cc-8658-48934d214f4b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:09:19.568250 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.568226 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "42e9614b-0672-41cc-8658-48934d214f4b" (UID: "42e9614b-0672-41cc-8658-48934d214f4b"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:09:19.568366 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.568244 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-online-tls" (OuterVolumeSpecName: "online-tls") pod "42e9614b-0672-41cc-8658-48934d214f4b" (UID: "42e9614b-0672-41cc-8658-48934d214f4b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:09:19.568366 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.568308 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e9614b-0672-41cc-8658-48934d214f4b-kube-api-access-kg5pt" (OuterVolumeSpecName: "kube-api-access-kg5pt") pod "42e9614b-0672-41cc-8658-48934d214f4b" (UID: "42e9614b-0672-41cc-8658-48934d214f4b"). InnerVolumeSpecName "kube-api-access-kg5pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:09:19.667170 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.667141 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg5pt\" (UniqueName: \"kubernetes.io/projected/42e9614b-0672-41cc-8658-48934d214f4b-kube-api-access-kg5pt\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:09:19.667170 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.667169 2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-offline-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:09:19.667349 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.667181 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:09:19.667349 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.667191 2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-ui-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:09:19.667349 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.667201 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/42e9614b-0672-41cc-8658-48934d214f4b-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:09:19.667349 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:19.667210 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/42e9614b-0672-41cc-8658-48934d214f4b-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:09:20.247031 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:20.246993 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" event={"ID":"42e9614b-0672-41cc-8658-48934d214f4b","Type":"ContainerDied","Data":"66cff2d32eeb133fa86643f569b23528a64ff8449a2e60a537ec0ea32c0ea9b0"} Sep 07 11:09:20.247398 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:20.247054 2571 scope.go:117] "RemoveContainer" containerID="94089da790aadcc1da720b5c2de9928089a88cb579e49bb5ab1f69ff7dc03a12" Sep 07 11:09:20.247398 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:20.247054 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5" Sep 07 11:09:20.255488 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:20.255466 2571 scope.go:117] "RemoveContainer" containerID="9fc728931cda994cc57c9c17ca0b4c61fc1d13a0fba0ecf22e81e0b574e821f0" Sep 07 11:09:20.263210 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:20.263174 2571 scope.go:117] "RemoveContainer" containerID="784b13551d36aeba410f235c5d12f7f2886687b5aa38f6accf1d44fed1b183fe" Sep 07 11:09:20.265711 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:20.265689 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5"] Sep 07 11:09:20.271645 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:20.271619 2571 scope.go:117] "RemoveContainer" containerID="03d1b2a81c328a7dd7d619852e51e76a3acbbcd2affcb9609e1e3bff5723c485" Sep 07 11:09:20.272111 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:20.272088 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-z2cv5"] Sep 07 11:09:20.278510 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:20.278487 2571 scope.go:117] "RemoveContainer" containerID="41778fb220c03f34d756ed91e202ded8e72d18ed67398ebee03c416002f60890" Sep 07 11:09:20.285098 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:20.285081 2571 scope.go:117] "RemoveContainer" containerID="26f6d2c53d23b622b40c4e37b28e36e6848ccee05b535a15017337ffb17bbe6c" Sep 07 11:09:22.123168 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:09:22.123134 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e9614b-0672-41cc-8658-48934d214f4b" path="/var/lib/kubelet/pods/42e9614b-0672-41cc-8658-48934d214f4b/volumes" Sep 07 11:10:24.605708 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.605676 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s"] Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.605992 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="ui" Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606003 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="ui" Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606026 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="feast-init" Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606031 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="feast-init" Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606037 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="feast-apply" Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606043 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="feast-apply" Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606050 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="registry" Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606054 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="registry" Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606060 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="offline" Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606065 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="offline" Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606071 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="online" Sep 07 11:10:24.606119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606076 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="online" Sep 07 11:10:24.606462 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606141 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="registry" Sep 07 11:10:24.606462 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606150 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="ui" Sep 07 11:10:24.606462 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606161 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="offline" Sep 07 11:10:24.606462 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.606167 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e9614b-0672-41cc-8658-48934d214f4b" containerName="online" Sep 07 11:10:24.609333 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.609317 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.611852 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.611826 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 07 11:10:24.612006 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.611863 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 07 11:10:24.612006 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.611912 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 07 11:10:24.612220 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.612140 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-v74g8\"" Sep 07 11:10:24.612220 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.612159 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 07 11:10:24.612220 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.612150 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 07 11:10:24.612894 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.612880 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 07 11:10:24.612975 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.612886 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 07 11:10:24.619111 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.619091 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s"] Sep 07 11:10:24.782393 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.782360 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-offline-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.782579 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.782428 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzndz\" (UniqueName: \"kubernetes.io/projected/b8747ccf-77f5-4991-a311-9d557fee92c5-kube-api-access-fzndz\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.782579 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.782451 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b8747ccf-77f5-4991-a311-9d557fee92c5-registry-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.782579 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.782467 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-online-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.782579 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.782567 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-ui-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.782718 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.782612 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8747ccf-77f5-4991-a311-9d557fee92c5-feast-data\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.883847 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.883756 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzndz\" (UniqueName: \"kubernetes.io/projected/b8747ccf-77f5-4991-a311-9d557fee92c5-kube-api-access-fzndz\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.883847 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.883801 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b8747ccf-77f5-4991-a311-9d557fee92c5-registry-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.883847 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.883820 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-online-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.884084 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.883869 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-ui-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.884084 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.883919 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8747ccf-77f5-4991-a311-9d557fee92c5-feast-data\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.884084 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.883955 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-offline-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.884084 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:10:24.884009 2571 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 07 11:10:24.884084 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:10:24.884076 2571 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 07 11:10:24.884252 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:10:24.884090 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-online-tls podName:b8747ccf-77f5-4991-a311-9d557fee92c5 nodeName:}" failed. No retries permitted until 2026-09-07 11:10:25.384071321 +0000 UTC m=+597.858427678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-online-tls") pod "feast-simple-feast-remote-setup-69bbbdf7bf-j245s" (UID: "b8747ccf-77f5-4991-a311-9d557fee92c5") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 07 11:10:24.884252 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:10:24.884128 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-offline-tls podName:b8747ccf-77f5-4991-a311-9d557fee92c5 nodeName:}" failed. No retries permitted until 2026-09-07 11:10:25.384112118 +0000 UTC m=+597.858468478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-offline-tls") pod "feast-simple-feast-remote-setup-69bbbdf7bf-j245s" (UID: "b8747ccf-77f5-4991-a311-9d557fee92c5") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 07 11:10:24.884326 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.884276 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8747ccf-77f5-4991-a311-9d557fee92c5-feast-data\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.884503 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.884481 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b8747ccf-77f5-4991-a311-9d557fee92c5-registry-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.886243 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.886218 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-ui-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:24.891850 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:24.891825 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzndz\" (UniqueName: \"kubernetes.io/projected/b8747ccf-77f5-4991-a311-9d557fee92c5-kube-api-access-fzndz\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:25.387775 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:25.387737 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-offline-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:25.387943 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:25.387813 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-online-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:25.390122 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:25.390095 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-offline-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:25.390258 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:25.390131 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-online-tls\") pod \"feast-simple-feast-remote-setup-69bbbdf7bf-j245s\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:25.518835 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:25.518800 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:25.794293 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:25.794269 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s"] Sep 07 11:10:25.795706 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:10:25.795683 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8747ccf_77f5_4991_a311_9d557fee92c5.slice/crio-591b35a8cc8fb528550c74d7b8f92859e3b9f3c03cde46a9d64a331891a3e764 WatchSource:0}: Error finding container 591b35a8cc8fb528550c74d7b8f92859e3b9f3c03cde46a9d64a331891a3e764: Status 404 returned error can't find the container with id 591b35a8cc8fb528550c74d7b8f92859e3b9f3c03cde46a9d64a331891a3e764 Sep 07 11:10:26.501002 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:26.500964 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerStarted","Data":"21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636"} Sep 07 11:10:26.501199 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:26.501016 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerStarted","Data":"591b35a8cc8fb528550c74d7b8f92859e3b9f3c03cde46a9d64a331891a3e764"} Sep 07 11:10:27.975208 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:27.975178 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:10:27.979943 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:27.979919 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:10:31.522490 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:31.522455 2571 generic.go:356] "Generic (PLEG): container finished" podID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerID="21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636" exitCode=0 Sep 07 11:10:31.522920 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:31.522529 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerDied","Data":"21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636"} Sep 07 11:10:32.528087 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:32.528050 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerStarted","Data":"53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912"} Sep 07 11:10:51.605679 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:51.605648 2571 generic.go:356] "Generic (PLEG): container finished" podID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerID="53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912" exitCode=0 Sep 07 11:10:51.606076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:51.605721 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerDied","Data":"53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912"} Sep 07 11:10:51.609034 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:51.609017 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 07 11:10:52.614592 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:52.614537 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerStarted","Data":"b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b"} Sep 07 11:10:52.614968 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:52.614614 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerStarted","Data":"ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6"} Sep 07 11:10:52.614968 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:52.614636 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerStarted","Data":"d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec"} Sep 07 11:10:52.638186 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:52.638127 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podStartSLOduration=28.638107421 podStartE2EDuration="28.638107421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:10:52.635389643 +0000 UTC m=+625.109746019" watchObservedRunningTime="2026-09-07 11:10:52.638107421 +0000 UTC m=+625.112463801" Sep 07 11:10:55.519381 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:55.519341 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:55.519938 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:55.519910 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:55.520046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:55.519944 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:55.520046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:55.519962 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:55.520046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:55.519978 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:55.520046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:55.519994 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:10:55.521358 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:55.521319 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.24:8443: connect: connection refused" Sep 07 11:10:55.521469 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:55.521366 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="online" probeResult="failure" output="Get \"https://10.132.0.24:6567/health\": dial tcp 10.132.0.24:6567: connect: connection refused" Sep 07 11:10:55.521469 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:55.521330 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.24:8816: connect: connection refused" Sep 07 11:10:58.519565 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:58.519504 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.24:8816: connect: connection refused" Sep 07 11:10:58.520140 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:58.519504 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.24:8443: connect: connection refused" Sep 07 11:10:58.520140 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:10:58.519585 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="online" probeResult="failure" output="Get \"https://10.132.0.24:6567/health\": dial tcp 10.132.0.24:6567: connect: connection refused" Sep 07 11:11:01.520821 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:11:01.520788 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:11:01.521197 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:11:01.520975 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:11:01.521726 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:11:01.521701 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:11:01.521962 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:11:01.521943 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:11:01.540204 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:11:01.540177 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:11:01.673994 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:11:01.673962 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:12:02.977158 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:02.977125 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s"] Sep 07 11:12:02.977827 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:02.977359 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="online" containerID="cri-o://d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec" gracePeriod=30 Sep 07 11:12:02.977827 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:02.977407 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="offline" containerID="cri-o://ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6" gracePeriod=30 Sep 07 11:12:02.977827 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:02.977431 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="ui" containerID="cri-o://b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b" gracePeriod=30 Sep 07 11:12:03.101202 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:03.101171 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s"] Sep 07 11:12:03.101446 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:03.101416 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="registry" containerID="cri-o://727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125" gracePeriod=30 Sep 07 11:12:03.101686 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:03.101441 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="ui" containerID="cri-o://66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1" gracePeriod=30 Sep 07 11:12:03.101800 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:03.101468 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="online" containerID="cri-o://7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e" gracePeriod=30 Sep 07 11:12:03.101800 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:03.101473 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="offline" containerID="cri-o://405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d" gracePeriod=30 Sep 07 11:12:03.897510 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:03.897482 2571 generic.go:356] "Generic (PLEG): container finished" podID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerID="b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b" exitCode=0 Sep 07 11:12:03.897510 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:03.897507 2571 generic.go:356] "Generic (PLEG): container finished" podID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerID="d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec" exitCode=0 Sep 07 11:12:03.897686 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:03.897561 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerDied","Data":"b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b"} Sep 07 11:12:03.897686 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:03.897608 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerDied","Data":"d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec"} Sep 07 11:12:03.900209 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:03.900186 2571 generic.go:356] "Generic (PLEG): container finished" podID="87641106-863f-412e-b8e2-cc370ec438ee" containerID="66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1" exitCode=0 Sep 07 11:12:03.900297 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:03.900263 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerDied","Data":"66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1"} Sep 07 11:12:04.907163 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:04.907130 2571 generic.go:356] "Generic (PLEG): container finished" podID="87641106-863f-412e-b8e2-cc370ec438ee" containerID="7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e" exitCode=0 Sep 07 11:12:04.907516 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:04.907197 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerDied","Data":"7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e"} Sep 07 11:12:11.143795 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:11.143745 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.23:8443: connect: connection refused" Sep 07 11:12:11.207645 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:11.207608 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="online" probeResult="failure" output="Get \"https://10.132.0.23:6567/health\": dial tcp 10.132.0.23:6567: connect: connection refused" Sep 07 11:12:11.521702 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:11.521664 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.24:8443: connect: connection refused" Sep 07 11:12:11.657602 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:11.657535 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="online" probeResult="failure" output="Get \"https://10.132.0.24:6567/health\": dial tcp 10.132.0.24:6567: connect: connection refused" Sep 07 11:12:21.143874 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:21.143776 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.23:8443: connect: connection refused" Sep 07 11:12:21.207364 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:21.207325 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="online" probeResult="failure" output="Get \"https://10.132.0.23:6567/health\": dial tcp 10.132.0.23:6567: connect: connection refused" Sep 07 11:12:21.522387 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:21.522349 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.24:8443: connect: connection refused" Sep 07 11:12:21.658236 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:21.658188 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="online" probeResult="failure" output="Get \"https://10.132.0.24:6567/health\": dial tcp 10.132.0.24:6567: connect: connection refused" Sep 07 11:12:31.143717 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:31.143667 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.23:8443: connect: connection refused" Sep 07 11:12:31.144174 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:31.143808 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:12:31.207420 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:31.207373 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="online" probeResult="failure" output="Get \"https://10.132.0.23:6567/health\": dial tcp 10.132.0.23:6567: connect: connection refused" Sep 07 11:12:31.207606 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:31.207486 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:12:31.522439 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:31.522398 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.24:8443: connect: connection refused" Sep 07 11:12:31.522637 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:31.522528 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:12:31.657773 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:31.657735 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="online" probeResult="failure" output="Get \"https://10.132.0.24:6567/health\": dial tcp 10.132.0.24:6567: connect: connection refused" Sep 07 11:12:31.657945 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:31.657843 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:12:33.243925 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.243846 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:12:33.425656 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.425604 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b8747ccf-77f5-4991-a311-9d557fee92c5-registry-tls\") pod \"b8747ccf-77f5-4991-a311-9d557fee92c5\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " Sep 07 11:12:33.425656 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.425665 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-offline-tls\") pod \"b8747ccf-77f5-4991-a311-9d557fee92c5\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " Sep 07 11:12:33.425901 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.425696 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzndz\" (UniqueName: \"kubernetes.io/projected/b8747ccf-77f5-4991-a311-9d557fee92c5-kube-api-access-fzndz\") pod \"b8747ccf-77f5-4991-a311-9d557fee92c5\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " Sep 07 11:12:33.425901 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.425752 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8747ccf-77f5-4991-a311-9d557fee92c5-feast-data\") pod \"b8747ccf-77f5-4991-a311-9d557fee92c5\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " Sep 07 11:12:33.425901 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.425780 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-online-tls\") pod \"b8747ccf-77f5-4991-a311-9d557fee92c5\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " Sep 07 11:12:33.425901 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.425821 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-ui-tls\") pod \"b8747ccf-77f5-4991-a311-9d557fee92c5\" (UID: \"b8747ccf-77f5-4991-a311-9d557fee92c5\") " Sep 07 11:12:33.426172 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.426054 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8747ccf-77f5-4991-a311-9d557fee92c5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b8747ccf-77f5-4991-a311-9d557fee92c5" (UID: "b8747ccf-77f5-4991-a311-9d557fee92c5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 11:12:33.426470 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.426389 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8747ccf-77f5-4991-a311-9d557fee92c5-feast-data" (OuterVolumeSpecName: "feast-data") pod "b8747ccf-77f5-4991-a311-9d557fee92c5" (UID: "b8747ccf-77f5-4991-a311-9d557fee92c5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:12:33.427988 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.427955 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b8747ccf-77f5-4991-a311-9d557fee92c5" (UID: "b8747ccf-77f5-4991-a311-9d557fee92c5"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:12:33.428080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.427996 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "b8747ccf-77f5-4991-a311-9d557fee92c5" (UID: "b8747ccf-77f5-4991-a311-9d557fee92c5"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:12:33.428080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.428023 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-online-tls" (OuterVolumeSpecName: "online-tls") pod "b8747ccf-77f5-4991-a311-9d557fee92c5" (UID: "b8747ccf-77f5-4991-a311-9d557fee92c5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:12:33.428080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.428032 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8747ccf-77f5-4991-a311-9d557fee92c5-kube-api-access-fzndz" (OuterVolumeSpecName: "kube-api-access-fzndz") pod "b8747ccf-77f5-4991-a311-9d557fee92c5" (UID: "b8747ccf-77f5-4991-a311-9d557fee92c5"). InnerVolumeSpecName "kube-api-access-fzndz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:12:33.526641 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.526590 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b8747ccf-77f5-4991-a311-9d557fee92c5-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:33.526641 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.526624 2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-offline-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:33.526641 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.526636 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzndz\" (UniqueName: \"kubernetes.io/projected/b8747ccf-77f5-4991-a311-9d557fee92c5-kube-api-access-fzndz\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:33.526641 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.526647 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8747ccf-77f5-4991-a311-9d557fee92c5-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:33.526895 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.526659 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:33.526895 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.526668 2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b8747ccf-77f5-4991-a311-9d557fee92c5-ui-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:33.741596 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.741569 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:12:33.930125 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.930042 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-registry-tls\") pod \"87641106-863f-412e-b8e2-cc370ec438ee\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " Sep 07 11:12:33.930125 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.930081 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-online-tls\") pod \"87641106-863f-412e-b8e2-cc370ec438ee\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " Sep 07 11:12:33.930125 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.930113 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/87641106-863f-412e-b8e2-cc370ec438ee-feast-data\") pod \"87641106-863f-412e-b8e2-cc370ec438ee\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " Sep 07 11:12:33.930125 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.930128 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-ui-tls\") pod \"87641106-863f-412e-b8e2-cc370ec438ee\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " Sep 07 11:12:33.930430 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.930254 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqjqb\" (UniqueName: \"kubernetes.io/projected/87641106-863f-412e-b8e2-cc370ec438ee-kube-api-access-kqjqb\") pod \"87641106-863f-412e-b8e2-cc370ec438ee\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " Sep 07 11:12:33.930430 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.930324 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-offline-tls\") pod \"87641106-863f-412e-b8e2-cc370ec438ee\" (UID: \"87641106-863f-412e-b8e2-cc370ec438ee\") " Sep 07 11:12:33.930890 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.930823 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87641106-863f-412e-b8e2-cc370ec438ee-feast-data" (OuterVolumeSpecName: "feast-data") pod "87641106-863f-412e-b8e2-cc370ec438ee" (UID: "87641106-863f-412e-b8e2-cc370ec438ee"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:12:33.932270 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.932242 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "87641106-863f-412e-b8e2-cc370ec438ee" (UID: "87641106-863f-412e-b8e2-cc370ec438ee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:12:33.932364 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.932270 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-online-tls" (OuterVolumeSpecName: "online-tls") pod "87641106-863f-412e-b8e2-cc370ec438ee" (UID: "87641106-863f-412e-b8e2-cc370ec438ee"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:12:33.932364 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.932290 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "87641106-863f-412e-b8e2-cc370ec438ee" (UID: "87641106-863f-412e-b8e2-cc370ec438ee"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:12:33.932649 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.932630 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87641106-863f-412e-b8e2-cc370ec438ee-kube-api-access-kqjqb" (OuterVolumeSpecName: "kube-api-access-kqjqb") pod "87641106-863f-412e-b8e2-cc370ec438ee" (UID: "87641106-863f-412e-b8e2-cc370ec438ee"). InnerVolumeSpecName "kube-api-access-kqjqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:12:33.932649 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:33.932638 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "87641106-863f-412e-b8e2-cc370ec438ee" (UID: "87641106-863f-412e-b8e2-cc370ec438ee"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:12:34.021781 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.021744 2571 generic.go:356] "Generic (PLEG): container finished" podID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerID="ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6" exitCode=137 Sep 07 11:12:34.021933 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.021816 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" Sep 07 11:12:34.021933 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.021833 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerDied","Data":"ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6"} Sep 07 11:12:34.021933 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.021886 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s" event={"ID":"b8747ccf-77f5-4991-a311-9d557fee92c5","Type":"ContainerDied","Data":"591b35a8cc8fb528550c74d7b8f92859e3b9f3c03cde46a9d64a331891a3e764"} Sep 07 11:12:34.021933 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.021915 2571 scope.go:117] "RemoveContainer" containerID="b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b" Sep 07 11:12:34.024872 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.024849 2571 generic.go:356] "Generic (PLEG): container finished" podID="87641106-863f-412e-b8e2-cc370ec438ee" containerID="405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d" exitCode=137 Sep 07 11:12:34.024872 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.024869 2571 generic.go:356] "Generic (PLEG): container finished" podID="87641106-863f-412e-b8e2-cc370ec438ee" containerID="727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125" exitCode=137 Sep 07 11:12:34.025012 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.024921 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" Sep 07 11:12:34.025012 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.024923 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerDied","Data":"405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d"} Sep 07 11:12:34.025080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.025024 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerDied","Data":"727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125"} Sep 07 11:12:34.025080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.025046 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s" event={"ID":"87641106-863f-412e-b8e2-cc370ec438ee","Type":"ContainerDied","Data":"b844805cca40260f288df613c8ac88df2318058e77af4040f39c132edc510b8f"} Sep 07 11:12:34.030896 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.030849 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:34.030896 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.030875 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:34.030896 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.030888 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/87641106-863f-412e-b8e2-cc370ec438ee-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:34.030896 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.030899 2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-ui-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:34.031135 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.030912 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqjqb\" (UniqueName: \"kubernetes.io/projected/87641106-863f-412e-b8e2-cc370ec438ee-kube-api-access-kqjqb\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:34.031135 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.030925 2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/87641106-863f-412e-b8e2-cc370ec438ee-offline-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:12:34.032404 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.032385 2571 scope.go:117] "RemoveContainer" containerID="ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6" Sep 07 11:12:34.040325 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.040308 2571 scope.go:117] "RemoveContainer" containerID="d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec" Sep 07 11:12:34.049033 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.046569 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s"] Sep 07 11:12:34.051322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.051296 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-69bbbdf7bf-j245s"] Sep 07 11:12:34.053404 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.053387 2571 scope.go:117] "RemoveContainer" containerID="53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912" Sep 07 11:12:34.060656 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.060591 2571 scope.go:117] "RemoveContainer" containerID="21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636" Sep 07 11:12:34.063484 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.063449 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s"] Sep 07 11:12:34.066462 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.066442 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6bc7846c6c-m722s"] Sep 07 11:12:34.070104 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.070087 2571 scope.go:117] "RemoveContainer" containerID="b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b" Sep 07 11:12:34.070393 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:34.070371 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b\": container with ID starting with b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b not found: ID does not exist" containerID="b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b" Sep 07 11:12:34.070438 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.070403 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b"} err="failed to get container status \"b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b\": rpc error: code = NotFound desc = could not find container \"b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b\": container with ID starting with b5d7525a528eb96bafacf974df6c6741ffdd2c4741ea661994da72f88e20fd3b not found: ID does not exist" Sep 07 11:12:34.070438 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.070421 2571 scope.go:117] "RemoveContainer" containerID="ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6" Sep 07 11:12:34.070673 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:34.070656 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6\": container with ID starting with ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6 not found: ID does not exist" containerID="ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6" Sep 07 11:12:34.070723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.070677 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6"} err="failed to get container status \"ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6\": rpc error: code = NotFound desc = could not find container \"ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6\": container with ID starting with ad451d0ae78e9117d9e3bd5b257faaf931b4896b61509e492b8ac2be1a5195f6 not found: ID does not exist" Sep 07 11:12:34.070723 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.070691 2571 scope.go:117] "RemoveContainer" containerID="d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec" Sep 07 11:12:34.070927 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:34.070912 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec\": container with ID starting with d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec not found: ID does not exist" containerID="d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec" Sep 07 11:12:34.070971 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.070930 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec"} err="failed to get container status \"d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec\": rpc error: code = NotFound desc = could not find container \"d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec\": container with ID starting with d6942baf9831542f314f44a0eea97daec1c1b2f0ea83b578519b534d01422fec not found: ID does not exist" Sep 07 11:12:34.070971 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.070943 2571 scope.go:117] "RemoveContainer" containerID="53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912" Sep 07 11:12:34.071134 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:34.071119 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912\": container with ID starting with 53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912 not found: ID does not exist" containerID="53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912" Sep 07 11:12:34.071178 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.071138 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912"} err="failed to get container status \"53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912\": rpc error: code = NotFound desc = could not find container \"53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912\": container with ID starting with 53938ad3e28ba557f59040aad2797ef741993de43ed931b2021d31f63a5e8912 not found: ID does not exist" Sep 07 11:12:34.071178 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.071150 2571 scope.go:117] "RemoveContainer" containerID="21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636" Sep 07 11:12:34.071339 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:34.071325 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636\": container with ID starting with 21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636 not found: ID does not exist" containerID="21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636" Sep 07 11:12:34.071379 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.071341 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636"} err="failed to get container status \"21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636\": rpc error: code = NotFound desc = could not find container \"21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636\": container with ID starting with 21799d86a18eda8a4079af826a834f40a70c4846b3c39045a19cf8fa5160b636 not found: ID does not exist" Sep 07 11:12:34.071379 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.071352 2571 scope.go:117] "RemoveContainer" containerID="66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1" Sep 07 11:12:34.078192 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.078176 2571 scope.go:117] "RemoveContainer" containerID="405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d" Sep 07 11:12:34.085224 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.085207 2571 scope.go:117] "RemoveContainer" containerID="7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e" Sep 07 11:12:34.091954 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.091938 2571 scope.go:117] "RemoveContainer" containerID="727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125" Sep 07 11:12:34.098565 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.098529 2571 scope.go:117] "RemoveContainer" containerID="cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1" Sep 07 11:12:34.105176 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.105161 2571 scope.go:117] "RemoveContainer" containerID="241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681" Sep 07 11:12:34.111703 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.111683 2571 scope.go:117] "RemoveContainer" containerID="66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1" Sep 07 11:12:34.111978 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:34.111957 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1\": container with ID starting with 66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1 not found: ID does not exist" containerID="66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1" Sep 07 11:12:34.112045 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.111982 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1"} err="failed to get container status \"66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1\": rpc error: code = NotFound desc = could not find container \"66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1\": container with ID starting with 66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1 not found: ID does not exist" Sep 07 11:12:34.112045 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.112000 2571 scope.go:117] "RemoveContainer" containerID="405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d" Sep 07 11:12:34.112271 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:34.112254 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d\": container with ID starting with 405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d not found: ID does not exist" containerID="405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d" Sep 07 11:12:34.112310 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.112277 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d"} err="failed to get container status \"405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d\": rpc error: code = NotFound desc = could not find container \"405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d\": container with ID starting with 405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d not found: ID does not exist" Sep 07 11:12:34.112310 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.112293 2571 scope.go:117] "RemoveContainer" containerID="7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e" Sep 07 11:12:34.112523 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:34.112507 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e\": container with ID starting with 7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e not found: ID does not exist" containerID="7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e" Sep 07 11:12:34.112599 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.112526 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e"} err="failed to get container status \"7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e\": rpc error: code = NotFound desc = could not find container \"7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e\": container with ID starting with 7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e not found: ID does not exist" Sep 07 11:12:34.112599 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.112539 2571 scope.go:117] "RemoveContainer" containerID="727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125" Sep 07 11:12:34.112785 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:34.112766 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125\": container with ID starting with 727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125 not found: ID does not exist" containerID="727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125" Sep 07 11:12:34.112826 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.112789 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125"} err="failed to get container status \"727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125\": rpc error: code = NotFound desc = could not find container \"727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125\": container with ID starting with 727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125 not found: ID does not exist" Sep 07 11:12:34.112826 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.112803 2571 scope.go:117] "RemoveContainer" containerID="cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1" Sep 07 11:12:34.113026 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:34.113012 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1\": container with ID starting with cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1 not found: ID does not exist" containerID="cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1" Sep 07 11:12:34.113066 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.113029 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1"} err="failed to get container status \"cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1\": rpc error: code = NotFound desc = could not find container \"cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1\": container with ID starting with cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1 not found: ID does not exist" Sep 07 11:12:34.113066 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.113040 2571 scope.go:117] "RemoveContainer" containerID="241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681" Sep 07 11:12:34.113262 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:34.113246 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681\": container with ID starting with 241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681 not found: ID does not exist" containerID="241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681" Sep 07 11:12:34.113310 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.113263 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681"} err="failed to get container status \"241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681\": rpc error: code = NotFound desc = could not find container \"241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681\": container with ID starting with 241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681 not found: ID does not exist" Sep 07 11:12:34.113310 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.113277 2571 scope.go:117] "RemoveContainer" containerID="66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1" Sep 07 11:12:34.113490 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.113474 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1"} err="failed to get container status \"66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1\": rpc error: code = NotFound desc = could not find container \"66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1\": container with ID starting with 66b29cebc3161cf6aa89b87e1727ce4025eb8913c28a4f4a9027d2de22641df1 not found: ID does not exist" Sep 07 11:12:34.113490 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.113490 2571 scope.go:117] "RemoveContainer" containerID="405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d" Sep 07 11:12:34.113961 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.113943 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d"} err="failed to get container status \"405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d\": rpc error: code = NotFound desc = could not find container \"405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d\": container with ID starting with 405b20071f3ad2367c48a9e33c539461491659160e133bdda75557ac134f383d not found: ID does not exist" Sep 07 11:12:34.114021 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.113961 2571 scope.go:117] "RemoveContainer" containerID="7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e" Sep 07 11:12:34.114194 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.114175 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e"} err="failed to get container status \"7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e\": rpc error: code = NotFound desc = could not find container \"7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e\": container with ID starting with 7cb86633c2097919f273763ae967596859e4edfa66a1f9fb785f89391e0c5a5e not found: ID does not exist" Sep 07 11:12:34.114242 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.114194 2571 scope.go:117] "RemoveContainer" containerID="727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125" Sep 07 11:12:34.114381 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.114365 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125"} err="failed to get container status \"727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125\": rpc error: code = NotFound desc = could not find container \"727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125\": container with ID starting with 727caae633b6aebc80030fe261efb83a7d3710581eaac49d021fe84018ad5125 not found: ID does not exist" Sep 07 11:12:34.114422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.114381 2571 scope.go:117] "RemoveContainer" containerID="cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1" Sep 07 11:12:34.114653 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.114633 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1"} err="failed to get container status \"cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1\": rpc error: code = NotFound desc = could not find container \"cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1\": container with ID starting with cc2e8e627a1d326c65b55c51557e7098d19ec293a07bfe4e1f5505c5e049e3c1 not found: ID does not exist" Sep 07 11:12:34.114704 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.114653 2571 scope.go:117] "RemoveContainer" containerID="241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681" Sep 07 11:12:34.114849 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.114831 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681"} err="failed to get container status \"241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681\": rpc error: code = NotFound desc = could not find container \"241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681\": container with ID starting with 241ed9e7d82bdcb676e4ea4f1a57906240bc9d41496db1b4078cc24666972681 not found: ID does not exist" Sep 07 11:12:34.123490 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.123468 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87641106-863f-412e-b8e2-cc370ec438ee" path="/var/lib/kubelet/pods/87641106-863f-412e-b8e2-cc370ec438ee/volumes" Sep 07 11:12:34.124125 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:34.124110 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" path="/var/lib/kubelet/pods/b8747ccf-77f5-4991-a311-9d557fee92c5/volumes" Sep 07 11:12:45.954084 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954043 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-hmkd7"] Sep 07 11:12:45.954569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954453 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="feast-apply" Sep 07 11:12:45.954569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954469 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="feast-apply" Sep 07 11:12:45.954569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954484 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="feast-apply" Sep 07 11:12:45.954569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954493 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="feast-apply" Sep 07 11:12:45.954569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954505 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="ui" Sep 07 11:12:45.954569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954514 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="ui" Sep 07 11:12:45.954569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954531 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="offline" Sep 07 11:12:45.954569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954540 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="offline" Sep 07 11:12:45.954569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954564 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="feast-init" Sep 07 11:12:45.954569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954573 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="feast-init" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954586 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="online" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954594 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="online" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954610 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="offline" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954618 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="offline" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954627 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="registry" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954635 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="registry" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954648 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="online" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954656 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="online" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954666 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="ui" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954674 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="ui" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954682 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="feast-init" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954689 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="feast-init" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954761 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="ui" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954773 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="online" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954782 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="registry" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954790 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="offline" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954798 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="offline" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954806 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8747ccf-77f5-4991-a311-9d557fee92c5" containerName="online" Sep 07 11:12:45.955088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.954815 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="87641106-863f-412e-b8e2-cc370ec438ee" containerName="ui" Sep 07 11:12:45.959990 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.959969 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-hmkd7" Sep 07 11:12:45.962263 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.962239 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 07 11:12:45.962382 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.962296 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-qkr69\"" Sep 07 11:12:45.962382 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.962320 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 07 11:12:45.963938 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.963650 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 07 11:12:45.965396 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.965374 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-hmkd7"] Sep 07 11:12:45.984983 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.984953 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-xbk7p"] Sep 07 11:12:45.988593 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.988570 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-xbk7p" Sep 07 11:12:45.993908 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:45.993886 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-xbk7p"] Sep 07 11:12:46.019796 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.019765 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgd8\" (UniqueName: \"kubernetes.io/projected/fc8a96c1-515d-405c-ab1f-6634886a8e4e-kube-api-access-kjgd8\") pod \"redis-69bbd798b6-xbk7p\" (UID: \"fc8a96c1-515d-405c-ab1f-6634886a8e4e\") " pod="test-ns-feast/redis-69bbd798b6-xbk7p" Sep 07 11:12:46.019796 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.019797 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wks7m\" (UniqueName: \"kubernetes.io/projected/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-kube-api-access-wks7m\") pod \"postgres-864846d4f5-hmkd7\" (UID: \"b282bd58-b4c7-44d0-b4de-d74f05db3bc8\") " pod="test-ns-feast/postgres-864846d4f5-hmkd7" Sep 07 11:12:46.019994 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.019817 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-postgresdata\") pod \"postgres-864846d4f5-hmkd7\" (UID: \"b282bd58-b4c7-44d0-b4de-d74f05db3bc8\") " pod="test-ns-feast/postgres-864846d4f5-hmkd7" Sep 07 11:12:46.120297 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.120269 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgd8\" (UniqueName: \"kubernetes.io/projected/fc8a96c1-515d-405c-ab1f-6634886a8e4e-kube-api-access-kjgd8\") pod \"redis-69bbd798b6-xbk7p\" (UID: \"fc8a96c1-515d-405c-ab1f-6634886a8e4e\") " pod="test-ns-feast/redis-69bbd798b6-xbk7p" Sep 07 11:12:46.120297 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.120302 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wks7m\" (UniqueName: \"kubernetes.io/projected/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-kube-api-access-wks7m\") pod \"postgres-864846d4f5-hmkd7\" (UID: \"b282bd58-b4c7-44d0-b4de-d74f05db3bc8\") " pod="test-ns-feast/postgres-864846d4f5-hmkd7" Sep 07 11:12:46.120490 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.120322 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-postgresdata\") pod \"postgres-864846d4f5-hmkd7\" (UID: \"b282bd58-b4c7-44d0-b4de-d74f05db3bc8\") " pod="test-ns-feast/postgres-864846d4f5-hmkd7" Sep 07 11:12:46.120652 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.120636 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-postgresdata\") pod \"postgres-864846d4f5-hmkd7\" (UID: \"b282bd58-b4c7-44d0-b4de-d74f05db3bc8\") " pod="test-ns-feast/postgres-864846d4f5-hmkd7" Sep 07 11:12:46.128067 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.128043 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgd8\" (UniqueName: \"kubernetes.io/projected/fc8a96c1-515d-405c-ab1f-6634886a8e4e-kube-api-access-kjgd8\") pod \"redis-69bbd798b6-xbk7p\" (UID: \"fc8a96c1-515d-405c-ab1f-6634886a8e4e\") " pod="test-ns-feast/redis-69bbd798b6-xbk7p" Sep 07 11:12:46.128192 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.128127 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wks7m\" (UniqueName: \"kubernetes.io/projected/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-kube-api-access-wks7m\") pod \"postgres-864846d4f5-hmkd7\" (UID: \"b282bd58-b4c7-44d0-b4de-d74f05db3bc8\") " pod="test-ns-feast/postgres-864846d4f5-hmkd7" Sep 07 11:12:46.270407 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.270381 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-hmkd7" Sep 07 11:12:46.298184 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.298155 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-xbk7p" Sep 07 11:12:46.538121 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.538092 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-hmkd7"] Sep 07 11:12:46.540470 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:12:46.540443 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb282bd58_b4c7_44d0_b4de_d74f05db3bc8.slice/crio-b0e9c3da9d937706bed9bcb2377bc42eeee1e650ae492d31a24ef441eba6fbd1 WatchSource:0}: Error finding container b0e9c3da9d937706bed9bcb2377bc42eeee1e650ae492d31a24ef441eba6fbd1: Status 404 returned error can't find the container with id b0e9c3da9d937706bed9bcb2377bc42eeee1e650ae492d31a24ef441eba6fbd1 Sep 07 11:12:46.542273 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.542258 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 11:12:46.571243 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:46.571220 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-xbk7p"] Sep 07 11:12:46.573430 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:12:46.573407 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc8a96c1_515d_405c_ab1f_6634886a8e4e.slice/crio-72b32bdc989d4bc1a487d268c616f84d7288e2feff30214a636a611ec5bf2f49 WatchSource:0}: Error finding container 72b32bdc989d4bc1a487d268c616f84d7288e2feff30214a636a611ec5bf2f49: Status 404 returned error can't find the container with id 72b32bdc989d4bc1a487d268c616f84d7288e2feff30214a636a611ec5bf2f49 Sep 07 11:12:47.075506 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:47.075459 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-xbk7p" event={"ID":"fc8a96c1-515d-405c-ab1f-6634886a8e4e","Type":"ContainerStarted","Data":"72b32bdc989d4bc1a487d268c616f84d7288e2feff30214a636a611ec5bf2f49"} Sep 07 11:12:47.076760 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:47.076722 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-hmkd7" event={"ID":"b282bd58-b4c7-44d0-b4de-d74f05db3bc8","Type":"ContainerStarted","Data":"b0e9c3da9d937706bed9bcb2377bc42eeee1e650ae492d31a24ef441eba6fbd1"} Sep 07 11:12:49.086918 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:49.086875 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-xbk7p" event={"ID":"fc8a96c1-515d-405c-ab1f-6634886a8e4e","Type":"ContainerStarted","Data":"8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844"} Sep 07 11:12:49.102286 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:49.102232 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-xbk7p" podStartSLOduration=2.484886392 podStartE2EDuration="4.102210342s" totalImagesPullingTime="1.61732395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:12:49.101074449 +0000 UTC m=+741.575430837" watchObservedRunningTime="2026-09-07 11:12:49.102210342 +0000 UTC m=+741.576566721" Sep 07 11:12:51.096909 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:51.096881 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-hmkd7" event={"ID":"b282bd58-b4c7-44d0-b4de-d74f05db3bc8","Type":"ContainerStarted","Data":"e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9"} Sep 07 11:12:51.111798 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:51.111743 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-hmkd7" podStartSLOduration=1.997683367 podStartE2EDuration="6.111722739s" totalImagesPullingTime="4.114039372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:12:51.110937868 +0000 UTC m=+743.585294247" watchObservedRunningTime="2026-09-07 11:12:51.111722739 +0000 UTC m=+743.586079118" Sep 07 11:12:54.863539 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.863498 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-9bb69459b-85pcs"] Sep 07 11:12:54.869829 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.869808 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:54.872265 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.872227 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 07 11:12:54.872388 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.872228 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 07 11:12:54.872388 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.872292 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 07 11:12:54.872584 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.872567 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 07 11:12:54.872659 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.872568 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-mcbqc\"" Sep 07 11:12:54.877533 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.877515 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-9bb69459b-85pcs"] Sep 07 11:12:54.991985 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.991949 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcb2eda-2e99-453c-b50e-5e86f95b1300-feast-data\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:54.991985 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.991985 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-online-tls\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:54.992182 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.992016 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-registry-tls\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:54.992182 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.992076 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4nqp\" (UniqueName: \"kubernetes.io/projected/5dcb2eda-2e99-453c-b50e-5e86f95b1300-kube-api-access-r4nqp\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:54.992182 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:54.992123 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-offline-tls\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.092876 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.092837 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-registry-tls\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.092876 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.092881 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4nqp\" (UniqueName: \"kubernetes.io/projected/5dcb2eda-2e99-453c-b50e-5e86f95b1300-kube-api-access-r4nqp\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.093066 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.092934 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-offline-tls\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.093066 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.092980 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcb2eda-2e99-453c-b50e-5e86f95b1300-feast-data\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.093066 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.093000 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-online-tls\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.093202 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:55.093086 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Sep 07 11:12:55.093202 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:55.093100 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Sep 07 11:12:55.093202 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:55.093137 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-online-tls podName:5dcb2eda-2e99-453c-b50e-5e86f95b1300 nodeName:}" failed. No retries permitted until 2026-09-07 11:12:55.593121864 +0000 UTC m=+748.067478221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-online-tls") pod "feast-credit-scoring-9bb69459b-85pcs" (UID: "5dcb2eda-2e99-453c-b50e-5e86f95b1300") : secret "feast-credit-scoring-online-tls" not found Sep 07 11:12:55.093202 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:12:55.093151 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-offline-tls podName:5dcb2eda-2e99-453c-b50e-5e86f95b1300 nodeName:}" failed. No retries permitted until 2026-09-07 11:12:55.593143762 +0000 UTC m=+748.067500119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-offline-tls") pod "feast-credit-scoring-9bb69459b-85pcs" (UID: "5dcb2eda-2e99-453c-b50e-5e86f95b1300") : secret "feast-credit-scoring-offline-tls" not found Sep 07 11:12:55.093396 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.093348 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcb2eda-2e99-453c-b50e-5e86f95b1300-feast-data\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.095499 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.095475 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-registry-tls\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.101335 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.101313 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4nqp\" (UniqueName: \"kubernetes.io/projected/5dcb2eda-2e99-453c-b50e-5e86f95b1300-kube-api-access-r4nqp\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.598058 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.598022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-online-tls\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.598253 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.598089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-offline-tls\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.600425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.600400 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-offline-tls\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.600561 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.600449 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-online-tls\") pod \"feast-credit-scoring-9bb69459b-85pcs\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:55.779441 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:55.779402 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:12:56.053735 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:56.053705 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-9bb69459b-85pcs"] Sep 07 11:12:56.054762 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:12:56.054736 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcb2eda_2e99_453c_b50e_5e86f95b1300.slice/crio-b49e975b7bbcbd79644ea8cde12bdc1e66bdd0fd43a98800d81a36661e4c51a6 WatchSource:0}: Error finding container b49e975b7bbcbd79644ea8cde12bdc1e66bdd0fd43a98800d81a36661e4c51a6: Status 404 returned error can't find the container with id b49e975b7bbcbd79644ea8cde12bdc1e66bdd0fd43a98800d81a36661e4c51a6 Sep 07 11:12:56.117716 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:56.117687 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerStarted","Data":"74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375"} Sep 07 11:12:56.117802 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:56.117742 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerStarted","Data":"b49e975b7bbcbd79644ea8cde12bdc1e66bdd0fd43a98800d81a36661e4c51a6"} Sep 07 11:12:58.126770 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:58.126735 2571 generic.go:356] "Generic (PLEG): container finished" podID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerID="74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375" exitCode=0 Sep 07 11:12:58.127131 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:58.126774 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerDied","Data":"74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375"} Sep 07 11:12:59.131617 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:12:59.131576 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerStarted","Data":"7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5"} Sep 07 11:13:05.157287 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:05.157251 2571 generic.go:356] "Generic (PLEG): container finished" podID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerID="7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5" exitCode=0 Sep 07 11:13:05.157821 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:05.157327 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerDied","Data":"7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5"} Sep 07 11:13:06.165506 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:06.165466 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerStarted","Data":"5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b"} Sep 07 11:13:06.166010 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:06.165543 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerStarted","Data":"bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58"} Sep 07 11:13:06.166010 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:06.165612 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerStarted","Data":"8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990"} Sep 07 11:13:06.190373 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:06.190311 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podStartSLOduration=12.190294235 podStartE2EDuration="12.190294235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:13:06.188171831 +0000 UTC m=+758.662528209" watchObservedRunningTime="2026-09-07 11:13:06.190294235 +0000 UTC m=+758.664650612" Sep 07 11:13:07.779866 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:07.779753 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:13:07.779866 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:07.779840 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:13:07.779866 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:07.779868 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:13:07.781030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:07.780992 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.27:8816: connect: connection refused" Sep 07 11:13:07.781161 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:07.781038 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 07 11:13:07.781161 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:07.781039 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.27:6571: connect: connection refused" Sep 07 11:13:10.780094 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:10.780036 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.27:8816: connect: connection refused" Sep 07 11:13:10.780611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:10.780118 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 07 11:13:10.780611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:10.780035 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.27:6571: connect: connection refused" Sep 07 11:13:13.780772 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:13.780723 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.27:6571: connect: connection refused" Sep 07 11:13:13.781719 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:13.781694 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:13:13.782080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:13.782060 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:13:13.782487 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:13.782466 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:13:13.788814 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:13.788796 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:13:14.209870 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:14.209787 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:13:14.214177 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:14.214154 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:13:15.780495 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:15.780446 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:13:16.779914 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:16.779857 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.27:6571: connect: connection refused" Sep 07 11:13:19.780533 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:19.780500 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:13:19.781326 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:13:19.781306 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:14:22.958198 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:22.958163 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-9bb69459b-85pcs"] Sep 07 11:14:22.958708 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:22.958379 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="registry" containerID="cri-o://8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990" gracePeriod=30 Sep 07 11:14:22.958708 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:22.958425 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="online" containerID="cri-o://bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58" gracePeriod=30 Sep 07 11:14:22.958708 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:22.958424 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="offline" containerID="cri-o://5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b" gracePeriod=30 Sep 07 11:14:22.981326 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:22.981291 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-hmkd7"] Sep 07 11:14:22.981567 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:22.981472 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-hmkd7" podUID="b282bd58-b4c7-44d0-b4de-d74f05db3bc8" containerName="postgres" containerID="cri-o://e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9" gracePeriod=30 Sep 07 11:14:22.985084 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:22.985059 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-xbk7p"] Sep 07 11:14:22.985271 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:22.985248 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-xbk7p" podUID="fc8a96c1-515d-405c-ab1f-6634886a8e4e" containerName="redis" containerID="cri-o://8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844" gracePeriod=30 Sep 07 11:14:23.235948 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.235923 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-xbk7p" Sep 07 11:14:23.237883 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.237864 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-hmkd7" Sep 07 11:14:23.322750 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.322720 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjgd8\" (UniqueName: \"kubernetes.io/projected/fc8a96c1-515d-405c-ab1f-6634886a8e4e-kube-api-access-kjgd8\") pod \"fc8a96c1-515d-405c-ab1f-6634886a8e4e\" (UID: \"fc8a96c1-515d-405c-ab1f-6634886a8e4e\") " Sep 07 11:14:23.322920 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.322819 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-postgresdata\") pod \"b282bd58-b4c7-44d0-b4de-d74f05db3bc8\" (UID: \"b282bd58-b4c7-44d0-b4de-d74f05db3bc8\") " Sep 07 11:14:23.322920 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.322838 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wks7m\" (UniqueName: \"kubernetes.io/projected/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-kube-api-access-wks7m\") pod \"b282bd58-b4c7-44d0-b4de-d74f05db3bc8\" (UID: \"b282bd58-b4c7-44d0-b4de-d74f05db3bc8\") " Sep 07 11:14:23.324800 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.324764 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8a96c1-515d-405c-ab1f-6634886a8e4e-kube-api-access-kjgd8" (OuterVolumeSpecName: "kube-api-access-kjgd8") pod "fc8a96c1-515d-405c-ab1f-6634886a8e4e" (UID: "fc8a96c1-515d-405c-ab1f-6634886a8e4e"). InnerVolumeSpecName "kube-api-access-kjgd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:14:23.324920 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.324888 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-kube-api-access-wks7m" (OuterVolumeSpecName: "kube-api-access-wks7m") pod "b282bd58-b4c7-44d0-b4de-d74f05db3bc8" (UID: "b282bd58-b4c7-44d0-b4de-d74f05db3bc8"). InnerVolumeSpecName "kube-api-access-wks7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:14:23.351509 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.351472 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "b282bd58-b4c7-44d0-b4de-d74f05db3bc8" (UID: "b282bd58-b4c7-44d0-b4de-d74f05db3bc8"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:14:23.424155 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.424117 2571 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-postgresdata\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:14:23.424155 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.424145 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wks7m\" (UniqueName: \"kubernetes.io/projected/b282bd58-b4c7-44d0-b4de-d74f05db3bc8-kube-api-access-wks7m\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:14:23.424155 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.424154 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjgd8\" (UniqueName: \"kubernetes.io/projected/fc8a96c1-515d-405c-ab1f-6634886a8e4e-kube-api-access-kjgd8\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:14:23.495006 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.494922 2571 generic.go:356] "Generic (PLEG): container finished" podID="fc8a96c1-515d-405c-ab1f-6634886a8e4e" containerID="8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844" exitCode=0 Sep 07 11:14:23.495006 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.494979 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-xbk7p" Sep 07 11:14:23.495205 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.495010 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-xbk7p" event={"ID":"fc8a96c1-515d-405c-ab1f-6634886a8e4e","Type":"ContainerDied","Data":"8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844"} Sep 07 11:14:23.495205 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.495053 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-xbk7p" event={"ID":"fc8a96c1-515d-405c-ab1f-6634886a8e4e","Type":"ContainerDied","Data":"72b32bdc989d4bc1a487d268c616f84d7288e2feff30214a636a611ec5bf2f49"} Sep 07 11:14:23.495205 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.495073 2571 scope.go:117] "RemoveContainer" containerID="8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844" Sep 07 11:14:23.496311 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.496291 2571 generic.go:356] "Generic (PLEG): container finished" podID="b282bd58-b4c7-44d0-b4de-d74f05db3bc8" containerID="e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9" exitCode=0 Sep 07 11:14:23.496420 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.496336 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-hmkd7" Sep 07 11:14:23.496420 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.496368 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-hmkd7" event={"ID":"b282bd58-b4c7-44d0-b4de-d74f05db3bc8","Type":"ContainerDied","Data":"e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9"} Sep 07 11:14:23.496420 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.496395 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-hmkd7" event={"ID":"b282bd58-b4c7-44d0-b4de-d74f05db3bc8","Type":"ContainerDied","Data":"b0e9c3da9d937706bed9bcb2377bc42eeee1e650ae492d31a24ef441eba6fbd1"} Sep 07 11:14:23.503524 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.503508 2571 scope.go:117] "RemoveContainer" containerID="8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844" Sep 07 11:14:23.503803 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:14:23.503784 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844\": container with ID starting with 8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844 not found: ID does not exist" containerID="8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844" Sep 07 11:14:23.503897 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.503810 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844"} err="failed to get container status \"8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844\": rpc error: code = NotFound desc = could not find container \"8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844\": container with ID starting with 8561c498d48403394f7ec98e3e0d101ec112e4f40fa3e460209bdaf279acd844 not found: ID does not exist" Sep 07 11:14:23.503897 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.503825 2571 scope.go:117] "RemoveContainer" containerID="e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9" Sep 07 11:14:23.511729 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.511707 2571 scope.go:117] "RemoveContainer" containerID="e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9" Sep 07 11:14:23.511950 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:14:23.511933 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9\": container with ID starting with e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9 not found: ID does not exist" containerID="e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9" Sep 07 11:14:23.512017 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.511954 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9"} err="failed to get container status \"e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9\": rpc error: code = NotFound desc = could not find container \"e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9\": container with ID starting with e7552c98d0df20d5a0e8d734f1434ef0d0b5d6c89a8908467086817bb53958b9 not found: ID does not exist" Sep 07 11:14:23.519367 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.519333 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-xbk7p"] Sep 07 11:14:23.522603 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.522582 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-xbk7p"] Sep 07 11:14:23.532528 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.532503 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-hmkd7"] Sep 07 11:14:23.535522 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:23.535498 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-hmkd7"] Sep 07 11:14:24.123496 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:24.123422 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b282bd58-b4c7-44d0-b4de-d74f05db3bc8" path="/var/lib/kubelet/pods/b282bd58-b4c7-44d0-b4de-d74f05db3bc8/volumes" Sep 07 11:14:24.123852 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:24.123730 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8a96c1-515d-405c-ab1f-6634886a8e4e" path="/var/lib/kubelet/pods/fc8a96c1-515d-405c-ab1f-6634886a8e4e/volumes" Sep 07 11:14:24.210349 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:24.210305 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 07 11:14:24.503716 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:24.503631 2571 generic.go:356] "Generic (PLEG): container finished" podID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerID="bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58" exitCode=0 Sep 07 11:14:24.503866 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:24.503707 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerDied","Data":"bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58"} Sep 07 11:14:34.210062 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:34.210018 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 07 11:14:44.210003 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:44.209956 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" Sep 07 11:14:44.210475 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:44.210080 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:14:53.601787 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.601762 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:14:53.613968 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.613942 2571 generic.go:356] "Generic (PLEG): container finished" podID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerID="5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b" exitCode=137 Sep 07 11:14:53.613968 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.613964 2571 generic.go:356] "Generic (PLEG): container finished" podID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerID="8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990" exitCode=137 Sep 07 11:14:53.614113 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.613998 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" Sep 07 11:14:53.614113 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.614002 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerDied","Data":"5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b"} Sep 07 11:14:53.614176 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.614151 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerDied","Data":"8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990"} Sep 07 11:14:53.614176 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.614170 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9bb69459b-85pcs" event={"ID":"5dcb2eda-2e99-453c-b50e-5e86f95b1300","Type":"ContainerDied","Data":"b49e975b7bbcbd79644ea8cde12bdc1e66bdd0fd43a98800d81a36661e4c51a6"} Sep 07 11:14:53.614235 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.614191 2571 scope.go:117] "RemoveContainer" containerID="5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b" Sep 07 11:14:53.621928 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.621897 2571 scope.go:117] "RemoveContainer" containerID="bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58" Sep 07 11:14:53.630592 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.630570 2571 scope.go:117] "RemoveContainer" containerID="8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990" Sep 07 11:14:53.640666 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.640645 2571 scope.go:117] "RemoveContainer" containerID="7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5" Sep 07 11:14:53.647525 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.647508 2571 scope.go:117] "RemoveContainer" containerID="74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375" Sep 07 11:14:53.654096 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.654079 2571 scope.go:117] "RemoveContainer" containerID="5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b" Sep 07 11:14:53.654376 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:14:53.654355 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b\": container with ID starting with 5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b not found: ID does not exist" containerID="5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b" Sep 07 11:14:53.654420 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.654387 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b"} err="failed to get container status \"5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b\": rpc error: code = NotFound desc = could not find container \"5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b\": container with ID starting with 5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b not found: ID does not exist" Sep 07 11:14:53.654420 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.654403 2571 scope.go:117] "RemoveContainer" containerID="bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58" Sep 07 11:14:53.654677 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:14:53.654658 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58\": container with ID starting with bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58 not found: ID does not exist" containerID="bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58" Sep 07 11:14:53.654757 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.654684 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58"} err="failed to get container status \"bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58\": rpc error: code = NotFound desc = could not find container \"bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58\": container with ID starting with bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58 not found: ID does not exist" Sep 07 11:14:53.654757 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.654704 2571 scope.go:117] "RemoveContainer" containerID="8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990" Sep 07 11:14:53.654965 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:14:53.654947 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990\": container with ID starting with 8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990 not found: ID does not exist" containerID="8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990" Sep 07 11:14:53.655006 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.654971 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990"} err="failed to get container status \"8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990\": rpc error: code = NotFound desc = could not find container \"8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990\": container with ID starting with 8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990 not found: ID does not exist" Sep 07 11:14:53.655006 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.654985 2571 scope.go:117] "RemoveContainer" containerID="7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5" Sep 07 11:14:53.655233 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:14:53.655215 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5\": container with ID starting with 7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5 not found: ID does not exist" containerID="7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5" Sep 07 11:14:53.655324 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.655235 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5"} err="failed to get container status \"7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5\": rpc error: code = NotFound desc = could not find container \"7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5\": container with ID starting with 7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5 not found: ID does not exist" Sep 07 11:14:53.655324 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.655250 2571 scope.go:117] "RemoveContainer" containerID="74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375" Sep 07 11:14:53.655478 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:14:53.655462 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375\": container with ID starting with 74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375 not found: ID does not exist" containerID="74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375" Sep 07 11:14:53.655512 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.655484 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375"} err="failed to get container status \"74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375\": rpc error: code = NotFound desc = could not find container \"74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375\": container with ID starting with 74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375 not found: ID does not exist" Sep 07 11:14:53.655512 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.655498 2571 scope.go:117] "RemoveContainer" containerID="5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b" Sep 07 11:14:53.655835 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.655727 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b"} err="failed to get container status \"5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b\": rpc error: code = NotFound desc = could not find container \"5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b\": container with ID starting with 5df356d840dfa39999f613b683f1102cba153a6f95950ae7afef7fb18ee11d6b not found: ID does not exist" Sep 07 11:14:53.655835 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.655744 2571 scope.go:117] "RemoveContainer" containerID="bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58" Sep 07 11:14:53.655971 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.655952 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58"} err="failed to get container status \"bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58\": rpc error: code = NotFound desc = could not find container \"bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58\": container with ID starting with bc9946afa37801970cb4c2c06c260b64eb3ebb194240451d152d77e96682eb58 not found: ID does not exist" Sep 07 11:14:53.656008 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.655972 2571 scope.go:117] "RemoveContainer" containerID="8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990" Sep 07 11:14:53.656191 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.656176 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990"} err="failed to get container status \"8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990\": rpc error: code = NotFound desc = could not find container \"8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990\": container with ID starting with 8f06105e54dc4a8656ddab1ca322b8c2b6d838bbd3d6ab879a4c9e4e5462f990 not found: ID does not exist" Sep 07 11:14:53.656227 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.656192 2571 scope.go:117] "RemoveContainer" containerID="7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5" Sep 07 11:14:53.656430 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.656414 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5"} err="failed to get container status \"7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5\": rpc error: code = NotFound desc = could not find container \"7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5\": container with ID starting with 7edbdf2e85204a1a3493dda7d7cab1ff5cea2b092c5d28f79a17fc908cd2e5f5 not found: ID does not exist" Sep 07 11:14:53.656472 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.656430 2571 scope.go:117] "RemoveContainer" containerID="74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375" Sep 07 11:14:53.656654 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.656629 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375"} err="failed to get container status \"74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375\": rpc error: code = NotFound desc = could not find container \"74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375\": container with ID starting with 74961c5a75121c09d8980b7947753248ddb478d9c95e66ae92210c334c694375 not found: ID does not exist" Sep 07 11:14:53.658941 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.658923 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4nqp\" (UniqueName: \"kubernetes.io/projected/5dcb2eda-2e99-453c-b50e-5e86f95b1300-kube-api-access-r4nqp\") pod \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " Sep 07 11:14:53.659018 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.658957 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-online-tls\") pod \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " Sep 07 11:14:53.659018 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.658979 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-offline-tls\") pod \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " Sep 07 11:14:53.659112 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.659026 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-registry-tls\") pod \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " Sep 07 11:14:53.659112 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.659079 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcb2eda-2e99-453c-b50e-5e86f95b1300-feast-data\") pod \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\" (UID: \"5dcb2eda-2e99-453c-b50e-5e86f95b1300\") " Sep 07 11:14:53.660976 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.660946 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5dcb2eda-2e99-453c-b50e-5e86f95b1300" (UID: "5dcb2eda-2e99-453c-b50e-5e86f95b1300"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:14:53.661055 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.660965 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5dcb2eda-2e99-453c-b50e-5e86f95b1300" (UID: "5dcb2eda-2e99-453c-b50e-5e86f95b1300"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:14:53.661228 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.661206 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-online-tls" (OuterVolumeSpecName: "online-tls") pod "5dcb2eda-2e99-453c-b50e-5e86f95b1300" (UID: "5dcb2eda-2e99-453c-b50e-5e86f95b1300"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:14:53.661329 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.661268 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcb2eda-2e99-453c-b50e-5e86f95b1300-kube-api-access-r4nqp" (OuterVolumeSpecName: "kube-api-access-r4nqp") pod "5dcb2eda-2e99-453c-b50e-5e86f95b1300" (UID: "5dcb2eda-2e99-453c-b50e-5e86f95b1300"). InnerVolumeSpecName "kube-api-access-r4nqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:14:53.667832 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.667804 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcb2eda-2e99-453c-b50e-5e86f95b1300-feast-data" (OuterVolumeSpecName: "feast-data") pod "5dcb2eda-2e99-453c-b50e-5e86f95b1300" (UID: "5dcb2eda-2e99-453c-b50e-5e86f95b1300"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:14:53.760611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.760515 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:14:53.760611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.760564 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcb2eda-2e99-453c-b50e-5e86f95b1300-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:14:53.760611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.760576 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4nqp\" (UniqueName: \"kubernetes.io/projected/5dcb2eda-2e99-453c-b50e-5e86f95b1300-kube-api-access-r4nqp\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:14:53.760611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.760604 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:14:53.760611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.760612 2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5dcb2eda-2e99-453c-b50e-5e86f95b1300-offline-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:14:53.936359 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.936330 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-9bb69459b-85pcs"] Sep 07 11:14:53.939963 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:53.939939 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-9bb69459b-85pcs"] Sep 07 11:14:54.123418 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:14:54.123384 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" path="/var/lib/kubelet/pods/5dcb2eda-2e99-453c-b50e-5e86f95b1300/volumes" Sep 07 11:15:28.000904 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:15:28.000834 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:15:28.003522 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:15:28.003496 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:20:28.032321 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:20:28.032239 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:20:28.032919 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:20:28.032739 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:25:28.055337 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:25:28.055309 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:25:28.056600 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:25:28.056572 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:26:28.341003 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.340928 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5dbqd"] Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341219 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="offline" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341230 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="offline" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341240 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8a96c1-515d-405c-ab1f-6634886a8e4e" containerName="redis" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341245 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8a96c1-515d-405c-ab1f-6634886a8e4e" containerName="redis" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341253 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="feast-apply" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341258 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="feast-apply" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341266 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b282bd58-b4c7-44d0-b4de-d74f05db3bc8" containerName="postgres" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341271 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b282bd58-b4c7-44d0-b4de-d74f05db3bc8" containerName="postgres" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341278 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="registry" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341283 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="registry" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341290 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="feast-init" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341295 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="feast-init" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341301 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="online" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341306 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="online" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341349 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="online" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341357 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8a96c1-515d-405c-ab1f-6634886a8e4e" containerName="redis" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341363 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b282bd58-b4c7-44d0-b4de-d74f05db3bc8" containerName="postgres" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341369 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="registry" Sep 07 11:26:28.341412 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.341376 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dcb2eda-2e99-453c-b50e-5e86f95b1300" containerName="offline" Sep 07 11:26:28.344318 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.344302 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" Sep 07 11:26:28.347163 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.347133 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 07 11:26:28.347279 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.347137 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-28lbh\"" Sep 07 11:26:28.348425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.348405 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 07 11:26:28.357365 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.357347 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5dbqd"] Sep 07 11:26:28.379678 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.379651 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8rhd\" (UniqueName: \"kubernetes.io/projected/0d33a8e0-0074-4449-87e5-397758923bf7-kube-api-access-t8rhd\") pod \"redis-69bbd798b6-5dbqd\" (UID: \"0d33a8e0-0074-4449-87e5-397758923bf7\") " pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" Sep 07 11:26:28.480641 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.480615 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8rhd\" (UniqueName: \"kubernetes.io/projected/0d33a8e0-0074-4449-87e5-397758923bf7-kube-api-access-t8rhd\") pod \"redis-69bbd798b6-5dbqd\" (UID: \"0d33a8e0-0074-4449-87e5-397758923bf7\") " pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" Sep 07 11:26:28.489362 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.489340 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8rhd\" (UniqueName: \"kubernetes.io/projected/0d33a8e0-0074-4449-87e5-397758923bf7-kube-api-access-t8rhd\") pod \"redis-69bbd798b6-5dbqd\" (UID: \"0d33a8e0-0074-4449-87e5-397758923bf7\") " pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" Sep 07 11:26:28.653317 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.653254 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" Sep 07 11:26:28.919801 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:28.919775 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5dbqd"] Sep 07 11:26:28.921679 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:26:28.921653 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d33a8e0_0074_4449_87e5_397758923bf7.slice/crio-bf14ac59fd08c72f5eca6c49a734f67d49506452398c7006fb8067f2667006b5 WatchSource:0}: Error finding container bf14ac59fd08c72f5eca6c49a734f67d49506452398c7006fb8067f2667006b5: Status 404 returned error can't find the container with id bf14ac59fd08c72f5eca6c49a734f67d49506452398c7006fb8067f2667006b5 Sep 07 11:26:29.128823 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:29.128786 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" event={"ID":"0d33a8e0-0074-4449-87e5-397758923bf7","Type":"ContainerStarted","Data":"6aca7bf9a1678e12328560a8237a904f74a28545fe0930866056ebaf75c837d1"} Sep 07 11:26:29.128823 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:29.128826 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" event={"ID":"0d33a8e0-0074-4449-87e5-397758923bf7","Type":"ContainerStarted","Data":"bf14ac59fd08c72f5eca6c49a734f67d49506452398c7006fb8067f2667006b5"} Sep 07 11:26:29.143689 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:29.143644 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" podStartSLOduration=1.143630691 podStartE2EDuration="1.143630691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:26:29.142759858 +0000 UTC m=+1561.617116236" watchObservedRunningTime="2026-09-07 11:26:29.143630691 +0000 UTC m=+1561.617987068" Sep 07 11:26:33.790602 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:33.790562 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4tjb4"] Sep 07 11:26:33.793933 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:33.793908 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" Sep 07 11:26:33.796745 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:33.796724 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 07 11:26:33.801537 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:33.801517 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4tjb4"] Sep 07 11:26:33.813827 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:33.813805 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ebb048d7-31f0-40e7-84ec-9bfacf57df36-postgresdata\") pod \"postgres-864846d4f5-4tjb4\" (UID: \"ebb048d7-31f0-40e7-84ec-9bfacf57df36\") " pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" Sep 07 11:26:33.813922 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:33.813872 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57cfk\" (UniqueName: \"kubernetes.io/projected/ebb048d7-31f0-40e7-84ec-9bfacf57df36-kube-api-access-57cfk\") pod \"postgres-864846d4f5-4tjb4\" (UID: \"ebb048d7-31f0-40e7-84ec-9bfacf57df36\") " pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" Sep 07 11:26:33.914519 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:33.914480 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ebb048d7-31f0-40e7-84ec-9bfacf57df36-postgresdata\") pod \"postgres-864846d4f5-4tjb4\" (UID: \"ebb048d7-31f0-40e7-84ec-9bfacf57df36\") " pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" Sep 07 11:26:33.914680 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:33.914533 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57cfk\" (UniqueName: \"kubernetes.io/projected/ebb048d7-31f0-40e7-84ec-9bfacf57df36-kube-api-access-57cfk\") pod \"postgres-864846d4f5-4tjb4\" (UID: \"ebb048d7-31f0-40e7-84ec-9bfacf57df36\") " pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" Sep 07 11:26:33.914866 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:33.914847 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ebb048d7-31f0-40e7-84ec-9bfacf57df36-postgresdata\") pod \"postgres-864846d4f5-4tjb4\" (UID: \"ebb048d7-31f0-40e7-84ec-9bfacf57df36\") " pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" Sep 07 11:26:33.922851 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:33.922822 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57cfk\" (UniqueName: \"kubernetes.io/projected/ebb048d7-31f0-40e7-84ec-9bfacf57df36-kube-api-access-57cfk\") pod \"postgres-864846d4f5-4tjb4\" (UID: \"ebb048d7-31f0-40e7-84ec-9bfacf57df36\") " pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" Sep 07 11:26:34.104037 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:34.103960 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" Sep 07 11:26:34.371205 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:34.371098 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4tjb4"] Sep 07 11:26:34.373762 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:26:34.373737 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebb048d7_31f0_40e7_84ec_9bfacf57df36.slice/crio-a91246a135c4c8df3d55f87588b691557ada53b276ab32e0e91a3b0c93a02702 WatchSource:0}: Error finding container a91246a135c4c8df3d55f87588b691557ada53b276ab32e0e91a3b0c93a02702: Status 404 returned error can't find the container with id a91246a135c4c8df3d55f87588b691557ada53b276ab32e0e91a3b0c93a02702 Sep 07 11:26:35.151961 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:35.151911 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" event={"ID":"ebb048d7-31f0-40e7-84ec-9bfacf57df36","Type":"ContainerStarted","Data":"5c6d7c298d1ca23f844942ebf3d6749dc66195a24418d879c902508b8444df16"} Sep 07 11:26:35.151961 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:35.151958 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" event={"ID":"ebb048d7-31f0-40e7-84ec-9bfacf57df36","Type":"ContainerStarted","Data":"a91246a135c4c8df3d55f87588b691557ada53b276ab32e0e91a3b0c93a02702"} Sep 07 11:26:35.167048 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:35.166994 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" podStartSLOduration=2.16697832 podStartE2EDuration="2.16697832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:26:35.166819975 +0000 UTC m=+1567.641176353" watchObservedRunningTime="2026-09-07 11:26:35.16697832 +0000 UTC m=+1567.641334699" Sep 07 11:26:39.467141 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.467103 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt"] Sep 07 11:26:39.470826 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.470802 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:39.473512 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.473493 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 07 11:26:39.473630 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.473491 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 07 11:26:39.473630 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.473527 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 07 11:26:39.473630 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.473501 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-wq7x7\"" Sep 07 11:26:39.480597 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.480576 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt"] Sep 07 11:26:39.550928 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.550906 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2m2r\" (UniqueName: \"kubernetes.io/projected/66645353-b1b3-4b61-953b-bdabd5178dc4-kube-api-access-v2m2r\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:39.551040 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.550961 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-registry-tls\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:39.551040 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.551000 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-online-tls\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:39.551040 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.551019 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/66645353-b1b3-4b61-953b-bdabd5178dc4-feast-data\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:39.651642 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.651618 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2m2r\" (UniqueName: \"kubernetes.io/projected/66645353-b1b3-4b61-953b-bdabd5178dc4-kube-api-access-v2m2r\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:39.651760 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.651661 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-registry-tls\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:39.651760 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.651686 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-online-tls\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:39.651760 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.651703 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/66645353-b1b3-4b61-953b-bdabd5178dc4-feast-data\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:39.651924 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:26:39.651793 2571 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Sep 07 11:26:39.651924 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:26:39.651856 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-online-tls podName:66645353-b1b3-4b61-953b-bdabd5178dc4 nodeName:}" failed. No retries permitted until 2026-09-07 11:26:40.151838455 +0000 UTC m=+1572.626194826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-online-tls") pod "feast-credit-scoring-6bd754576f-dhckt" (UID: "66645353-b1b3-4b61-953b-bdabd5178dc4") : secret "feast-credit-scoring-online-tls" not found Sep 07 11:26:39.652061 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.652042 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/66645353-b1b3-4b61-953b-bdabd5178dc4-feast-data\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:39.653978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.653962 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-registry-tls\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:39.660029 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:39.660012 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2m2r\" (UniqueName: \"kubernetes.io/projected/66645353-b1b3-4b61-953b-bdabd5178dc4-kube-api-access-v2m2r\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:40.155699 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:40.155667 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-online-tls\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:40.158010 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:40.157993 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-online-tls\") pod \"feast-credit-scoring-6bd754576f-dhckt\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:40.379413 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:40.379385 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:40.650007 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:40.649979 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt"] Sep 07 11:26:40.650614 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:26:40.650591 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66645353_b1b3_4b61_953b_bdabd5178dc4.slice/crio-b5162a49062e6156bd3ab3fdd06ced8994e675981463de33aece664448979fa1 WatchSource:0}: Error finding container b5162a49062e6156bd3ab3fdd06ced8994e675981463de33aece664448979fa1: Status 404 returned error can't find the container with id b5162a49062e6156bd3ab3fdd06ced8994e675981463de33aece664448979fa1 Sep 07 11:26:41.175119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:41.175071 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" event={"ID":"66645353-b1b3-4b61-953b-bdabd5178dc4","Type":"ContainerStarted","Data":"b7ed943d4a86c28e0d8564e87d6a46e9780b53ee23a52a0cb58384176f7e278b"} Sep 07 11:26:41.175298 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:41.175143 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" event={"ID":"66645353-b1b3-4b61-953b-bdabd5178dc4","Type":"ContainerStarted","Data":"b5162a49062e6156bd3ab3fdd06ced8994e675981463de33aece664448979fa1"} Sep 07 11:26:41.759801 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:26:41.759776 2571 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66645353_b1b3_4b61_953b_bdabd5178dc4.slice/crio-conmon-b7ed943d4a86c28e0d8564e87d6a46e9780b53ee23a52a0cb58384176f7e278b.scope\": RecentStats: unable to find data in memory cache]" Sep 07 11:26:42.179496 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:42.179463 2571 generic.go:356] "Generic (PLEG): container finished" podID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerID="b7ed943d4a86c28e0d8564e87d6a46e9780b53ee23a52a0cb58384176f7e278b" exitCode=0 Sep 07 11:26:42.179660 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:42.179570 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" event={"ID":"66645353-b1b3-4b61-953b-bdabd5178dc4","Type":"ContainerDied","Data":"b7ed943d4a86c28e0d8564e87d6a46e9780b53ee23a52a0cb58384176f7e278b"} Sep 07 11:26:43.185837 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:43.185796 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" event={"ID":"66645353-b1b3-4b61-953b-bdabd5178dc4","Type":"ContainerStarted","Data":"a76bff70b6c25c1a6d5938c4a6f804b5f33554d66e5675835d74033b4338b6b9"} Sep 07 11:26:49.209847 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:49.209758 2571 generic.go:356] "Generic (PLEG): container finished" podID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerID="a76bff70b6c25c1a6d5938c4a6f804b5f33554d66e5675835d74033b4338b6b9" exitCode=0 Sep 07 11:26:49.210282 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:49.209839 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" event={"ID":"66645353-b1b3-4b61-953b-bdabd5178dc4","Type":"ContainerDied","Data":"a76bff70b6c25c1a6d5938c4a6f804b5f33554d66e5675835d74033b4338b6b9"} Sep 07 11:26:50.216272 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:50.216235 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" event={"ID":"66645353-b1b3-4b61-953b-bdabd5178dc4","Type":"ContainerStarted","Data":"d0c65c4407ea720620cad3c2b3140e623a5349175cb2bc04416a971f3cd4f3fe"} Sep 07 11:26:50.216796 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:50.216288 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" event={"ID":"66645353-b1b3-4b61-953b-bdabd5178dc4","Type":"ContainerStarted","Data":"a89bc7c8c445627891d579eaaab2973e486beaead844598e6ce886c77cd7dcf7"} Sep 07 11:26:50.248160 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:50.248104 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" podStartSLOduration=11.248084769 podStartE2EDuration="11.248084769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:26:50.246339692 +0000 UTC m=+1582.720696094" watchObservedRunningTime="2026-09-07 11:26:50.248084769 +0000 UTC m=+1582.722441148" Sep 07 11:26:50.380408 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:50.380359 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:50.380579 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:50.380438 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:52.379907 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:52.379865 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:52.380385 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:52.379936 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:52.381644 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:52.381607 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" Sep 07 11:26:52.381764 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:52.381652 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 07 11:26:55.380330 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:55.380285 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" Sep 07 11:26:55.400291 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:55.400263 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:55.403643 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:55.403619 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:26:58.379799 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:26:58.379744 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" Sep 07 11:27:01.380766 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:01.380732 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:27:01.381443 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:01.381423 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:27:05.331202 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.331169 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl"] Sep 07 11:27:05.334798 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.334780 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:05.337463 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.337444 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-h9j58\"" Sep 07 11:27:05.337590 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.337441 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 07 11:27:05.337590 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.337445 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 07 11:27:05.337697 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.337655 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 07 11:27:05.346478 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.346456 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl"] Sep 07 11:27:05.468856 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.468824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssd4p\" (UniqueName: \"kubernetes.io/projected/561ba036-bb2d-49b5-91e2-0b252e71d3ed-kube-api-access-ssd4p\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:05.469016 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.468886 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/561ba036-bb2d-49b5-91e2-0b252e71d3ed-feast-data\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:05.469016 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.468926 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-online-tls\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:05.469016 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.468951 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-registry-tls\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:05.570354 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.570320 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/561ba036-bb2d-49b5-91e2-0b252e71d3ed-feast-data\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:05.570354 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.570359 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-online-tls\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:05.570596 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.570386 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-registry-tls\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:05.570596 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.570434 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssd4p\" (UniqueName: \"kubernetes.io/projected/561ba036-bb2d-49b5-91e2-0b252e71d3ed-kube-api-access-ssd4p\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:05.570596 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:27:05.570539 2571 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found Sep 07 11:27:05.570758 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:27:05.570628 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-registry-tls podName:561ba036-bb2d-49b5-91e2-0b252e71d3ed nodeName:}" failed. No retries permitted until 2026-09-07 11:27:06.070610161 +0000 UTC m=+1598.544966517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-registry-tls") pod "feast-driver-ranking-677444b594-rdpwl" (UID: "561ba036-bb2d-49b5-91e2-0b252e71d3ed") : secret "feast-driver-ranking-registry-tls" not found Sep 07 11:27:05.570825 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.570785 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/561ba036-bb2d-49b5-91e2-0b252e71d3ed-feast-data\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:05.572912 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.572887 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-online-tls\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:05.580626 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:05.580603 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssd4p\" (UniqueName: \"kubernetes.io/projected/561ba036-bb2d-49b5-91e2-0b252e71d3ed-kube-api-access-ssd4p\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:06.075496 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:06.075463 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-registry-tls\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:06.077967 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:06.077944 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-registry-tls\") pod \"feast-driver-ranking-677444b594-rdpwl\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:06.244113 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:06.244076 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:06.517333 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:06.517152 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl"] Sep 07 11:27:06.519944 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:27:06.519919 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561ba036_bb2d_49b5_91e2_0b252e71d3ed.slice/crio-b31452e74b7af804356d49d9528a55eaa95443b7dae17cd7a1bf48321538aff7 WatchSource:0}: Error finding container b31452e74b7af804356d49d9528a55eaa95443b7dae17cd7a1bf48321538aff7: Status 404 returned error can't find the container with id b31452e74b7af804356d49d9528a55eaa95443b7dae17cd7a1bf48321538aff7 Sep 07 11:27:07.291308 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:07.291259 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" event={"ID":"561ba036-bb2d-49b5-91e2-0b252e71d3ed","Type":"ContainerStarted","Data":"455a50c4db54ece558d6fa066625d05037ae46889f7e551c58f3edd56c1585fb"} Sep 07 11:27:07.291478 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:07.291332 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" event={"ID":"561ba036-bb2d-49b5-91e2-0b252e71d3ed","Type":"ContainerStarted","Data":"b31452e74b7af804356d49d9528a55eaa95443b7dae17cd7a1bf48321538aff7"} Sep 07 11:27:11.309519 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:11.309483 2571 generic.go:356] "Generic (PLEG): container finished" podID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerID="455a50c4db54ece558d6fa066625d05037ae46889f7e551c58f3edd56c1585fb" exitCode=0 Sep 07 11:27:11.309884 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:11.309566 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" event={"ID":"561ba036-bb2d-49b5-91e2-0b252e71d3ed","Type":"ContainerDied","Data":"455a50c4db54ece558d6fa066625d05037ae46889f7e551c58f3edd56c1585fb"} Sep 07 11:27:12.315140 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:12.315106 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" event={"ID":"561ba036-bb2d-49b5-91e2-0b252e71d3ed","Type":"ContainerStarted","Data":"18ea8c6fde455952407195e02d48fc99fb7f0caf7b96fe958389d48c888691a8"} Sep 07 11:27:18.345184 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:18.345151 2571 generic.go:356] "Generic (PLEG): container finished" podID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerID="18ea8c6fde455952407195e02d48fc99fb7f0caf7b96fe958389d48c888691a8" exitCode=0 Sep 07 11:27:18.345615 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:18.345239 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" event={"ID":"561ba036-bb2d-49b5-91e2-0b252e71d3ed","Type":"ContainerDied","Data":"18ea8c6fde455952407195e02d48fc99fb7f0caf7b96fe958389d48c888691a8"} Sep 07 11:27:19.353236 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:19.353200 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" event={"ID":"561ba036-bb2d-49b5-91e2-0b252e71d3ed","Type":"ContainerStarted","Data":"115bc6ddfacef37ef0824588ece426d6090a731be7a855845184e5e56143586f"} Sep 07 11:27:19.353736 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:19.353257 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" event={"ID":"561ba036-bb2d-49b5-91e2-0b252e71d3ed","Type":"ContainerStarted","Data":"642cf588ec6bc9a2e045d34f9f808aefd15aed77ab210c60f55830c6a7a74186"} Sep 07 11:27:19.374785 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:19.374727 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" podStartSLOduration=14.374709589 podStartE2EDuration="14.374709589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:27:19.374321661 +0000 UTC m=+1611.848678051" watchObservedRunningTime="2026-09-07 11:27:19.374709589 +0000 UTC m=+1611.849065974" Sep 07 11:27:21.245299 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:21.245207 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:21.245299 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:21.245283 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:21.246936 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:21.246901 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.31:6571: connect: connection refused" Sep 07 11:27:21.247027 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:21.246916 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 07 11:27:24.244808 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:24.244757 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 07 11:27:24.244808 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:24.244775 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.31:6571: connect: connection refused" Sep 07 11:27:26.244989 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:26.244950 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:26.245381 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:26.245027 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:27.245080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:27.245024 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.31:6571: connect: connection refused" Sep 07 11:27:27.251227 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:27.251196 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:27.254712 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:27.254687 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:30.245613 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:30.245582 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:27:30.246382 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:27:30.246362 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:28:49.137223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.137182 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5"] Sep 07 11:28:49.137794 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.137483 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" podUID="085001ba-6664-4434-be0b-fee420861e51" containerName="manager" containerID="cri-o://574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee" gracePeriod=10 Sep 07 11:28:49.404685 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.404605 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" Sep 07 11:28:49.517004 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.516963 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt"] Sep 07 11:28:49.517337 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.517274 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="registry" containerID="cri-o://a89bc7c8c445627891d579eaaab2973e486beaead844598e6ce886c77cd7dcf7" gracePeriod=30 Sep 07 11:28:49.517428 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.517303 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="online" containerID="cri-o://d0c65c4407ea720620cad3c2b3140e623a5349175cb2bc04416a971f3cd4f3fe" gracePeriod=30 Sep 07 11:28:49.531057 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.531018 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl"] Sep 07 11:28:49.531349 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.531317 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="registry" containerID="cri-o://642cf588ec6bc9a2e045d34f9f808aefd15aed77ab210c60f55830c6a7a74186" gracePeriod=30 Sep 07 11:28:49.531452 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.531351 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="online" containerID="cri-o://115bc6ddfacef37ef0824588ece426d6090a731be7a855845184e5e56143586f" gracePeriod=30 Sep 07 11:28:49.550520 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.550482 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5wlz\" (UniqueName: \"kubernetes.io/projected/085001ba-6664-4434-be0b-fee420861e51-kube-api-access-p5wlz\") pod \"085001ba-6664-4434-be0b-fee420861e51\" (UID: \"085001ba-6664-4434-be0b-fee420861e51\") " Sep 07 11:28:49.552677 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.552641 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085001ba-6664-4434-be0b-fee420861e51-kube-api-access-p5wlz" (OuterVolumeSpecName: "kube-api-access-p5wlz") pod "085001ba-6664-4434-be0b-fee420861e51" (UID: "085001ba-6664-4434-be0b-fee420861e51"). InnerVolumeSpecName "kube-api-access-p5wlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:28:49.651890 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.651846 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5wlz\" (UniqueName: \"kubernetes.io/projected/085001ba-6664-4434-be0b-fee420861e51-kube-api-access-p5wlz\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:28:49.736851 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.736767 2571 generic.go:356] "Generic (PLEG): container finished" podID="085001ba-6664-4434-be0b-fee420861e51" containerID="574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee" exitCode=0 Sep 07 11:28:49.736851 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.736807 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" Sep 07 11:28:49.737077 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.736851 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" event={"ID":"085001ba-6664-4434-be0b-fee420861e51","Type":"ContainerDied","Data":"574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee"} Sep 07 11:28:49.737077 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.736908 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5" event={"ID":"085001ba-6664-4434-be0b-fee420861e51","Type":"ContainerDied","Data":"70769682260ad09f463be47b8a7d2271f4204f3adcb649add528802a5fbed4c5"} Sep 07 11:28:49.737077 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.736937 2571 scope.go:117] "RemoveContainer" containerID="574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee" Sep 07 11:28:49.745611 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.745592 2571 scope.go:117] "RemoveContainer" containerID="574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee" Sep 07 11:28:49.745910 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:28:49.745892 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee\": container with ID starting with 574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee not found: ID does not exist" containerID="574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee" Sep 07 11:28:49.745957 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.745919 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee"} err="failed to get container status \"574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee\": rpc error: code = NotFound desc = could not find container \"574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee\": container with ID starting with 574e425cba47f2d8baba6d2d722bceb6c7bf10e24a4bbb19970cb26040cf74ee not found: ID does not exist" Sep 07 11:28:49.767025 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.766989 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5"] Sep 07 11:28:49.770177 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:49.770145 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7fd64c9d98-cg5p5"] Sep 07 11:28:50.125213 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.125169 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="085001ba-6664-4434-be0b-fee420861e51" path="/var/lib/kubelet/pods/085001ba-6664-4434-be0b-fee420861e51/volumes" Sep 07 11:28:50.553819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.553784 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4tjb4"] Sep 07 11:28:50.554240 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.553962 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" podUID="ebb048d7-31f0-40e7-84ec-9bfacf57df36" containerName="postgres" containerID="cri-o://5c6d7c298d1ca23f844942ebf3d6749dc66195a24418d879c902508b8444df16" gracePeriod=30 Sep 07 11:28:50.557952 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.557924 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5dbqd"] Sep 07 11:28:50.558118 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.558098 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" podUID="0d33a8e0-0074-4449-87e5-397758923bf7" containerName="redis" containerID="cri-o://6aca7bf9a1678e12328560a8237a904f74a28545fe0930866056ebaf75c837d1" gracePeriod=30 Sep 07 11:28:50.744250 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.744220 2571 generic.go:356] "Generic (PLEG): container finished" podID="0d33a8e0-0074-4449-87e5-397758923bf7" containerID="6aca7bf9a1678e12328560a8237a904f74a28545fe0930866056ebaf75c837d1" exitCode=0 Sep 07 11:28:50.744429 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.744289 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" event={"ID":"0d33a8e0-0074-4449-87e5-397758923bf7","Type":"ContainerDied","Data":"6aca7bf9a1678e12328560a8237a904f74a28545fe0930866056ebaf75c837d1"} Sep 07 11:28:50.745807 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.745786 2571 generic.go:356] "Generic (PLEG): container finished" podID="ebb048d7-31f0-40e7-84ec-9bfacf57df36" containerID="5c6d7c298d1ca23f844942ebf3d6749dc66195a24418d879c902508b8444df16" exitCode=0 Sep 07 11:28:50.745916 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.745862 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" event={"ID":"ebb048d7-31f0-40e7-84ec-9bfacf57df36","Type":"ContainerDied","Data":"5c6d7c298d1ca23f844942ebf3d6749dc66195a24418d879c902508b8444df16"} Sep 07 11:28:50.748246 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.748213 2571 generic.go:356] "Generic (PLEG): container finished" podID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerID="d0c65c4407ea720620cad3c2b3140e623a5349175cb2bc04416a971f3cd4f3fe" exitCode=0 Sep 07 11:28:50.748340 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.748284 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" event={"ID":"66645353-b1b3-4b61-953b-bdabd5178dc4","Type":"ContainerDied","Data":"d0c65c4407ea720620cad3c2b3140e623a5349175cb2bc04416a971f3cd4f3fe"} Sep 07 11:28:50.750666 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.750645 2571 generic.go:356] "Generic (PLEG): container finished" podID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerID="115bc6ddfacef37ef0824588ece426d6090a731be7a855845184e5e56143586f" exitCode=0 Sep 07 11:28:50.750758 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.750683 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" event={"ID":"561ba036-bb2d-49b5-91e2-0b252e71d3ed","Type":"ContainerDied","Data":"115bc6ddfacef37ef0824588ece426d6090a731be7a855845184e5e56143586f"} Sep 07 11:28:50.813601 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.813580 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" Sep 07 11:28:50.819866 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.819844 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" Sep 07 11:28:50.962574 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.962445 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ebb048d7-31f0-40e7-84ec-9bfacf57df36-postgresdata\") pod \"ebb048d7-31f0-40e7-84ec-9bfacf57df36\" (UID: \"ebb048d7-31f0-40e7-84ec-9bfacf57df36\") " Sep 07 11:28:50.962574 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.962499 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8rhd\" (UniqueName: \"kubernetes.io/projected/0d33a8e0-0074-4449-87e5-397758923bf7-kube-api-access-t8rhd\") pod \"0d33a8e0-0074-4449-87e5-397758923bf7\" (UID: \"0d33a8e0-0074-4449-87e5-397758923bf7\") " Sep 07 11:28:50.962574 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.962531 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57cfk\" (UniqueName: \"kubernetes.io/projected/ebb048d7-31f0-40e7-84ec-9bfacf57df36-kube-api-access-57cfk\") pod \"ebb048d7-31f0-40e7-84ec-9bfacf57df36\" (UID: \"ebb048d7-31f0-40e7-84ec-9bfacf57df36\") " Sep 07 11:28:50.964873 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.964828 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb048d7-31f0-40e7-84ec-9bfacf57df36-kube-api-access-57cfk" (OuterVolumeSpecName: "kube-api-access-57cfk") pod "ebb048d7-31f0-40e7-84ec-9bfacf57df36" (UID: "ebb048d7-31f0-40e7-84ec-9bfacf57df36"). InnerVolumeSpecName "kube-api-access-57cfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:28:50.965007 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.964872 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d33a8e0-0074-4449-87e5-397758923bf7-kube-api-access-t8rhd" (OuterVolumeSpecName: "kube-api-access-t8rhd") pod "0d33a8e0-0074-4449-87e5-397758923bf7" (UID: "0d33a8e0-0074-4449-87e5-397758923bf7"). InnerVolumeSpecName "kube-api-access-t8rhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:28:50.992226 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:50.992176 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb048d7-31f0-40e7-84ec-9bfacf57df36-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "ebb048d7-31f0-40e7-84ec-9bfacf57df36" (UID: "ebb048d7-31f0-40e7-84ec-9bfacf57df36"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:28:51.063655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.063613 2571 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ebb048d7-31f0-40e7-84ec-9bfacf57df36-postgresdata\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:28:51.063655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.063648 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8rhd\" (UniqueName: \"kubernetes.io/projected/0d33a8e0-0074-4449-87e5-397758923bf7-kube-api-access-t8rhd\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:28:51.063655 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.063660 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57cfk\" (UniqueName: \"kubernetes.io/projected/ebb048d7-31f0-40e7-84ec-9bfacf57df36-kube-api-access-57cfk\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:28:51.756255 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.756174 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" Sep 07 11:28:51.756710 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.756181 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-5dbqd" event={"ID":"0d33a8e0-0074-4449-87e5-397758923bf7","Type":"ContainerDied","Data":"bf14ac59fd08c72f5eca6c49a734f67d49506452398c7006fb8067f2667006b5"} Sep 07 11:28:51.756710 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.756304 2571 scope.go:117] "RemoveContainer" containerID="6aca7bf9a1678e12328560a8237a904f74a28545fe0930866056ebaf75c837d1" Sep 07 11:28:51.757574 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.757509 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" Sep 07 11:28:51.757574 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.757510 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4tjb4" event={"ID":"ebb048d7-31f0-40e7-84ec-9bfacf57df36","Type":"ContainerDied","Data":"a91246a135c4c8df3d55f87588b691557ada53b276ab32e0e91a3b0c93a02702"} Sep 07 11:28:51.764497 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.764390 2571 scope.go:117] "RemoveContainer" containerID="5c6d7c298d1ca23f844942ebf3d6749dc66195a24418d879c902508b8444df16" Sep 07 11:28:51.778485 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.778458 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4tjb4"] Sep 07 11:28:51.785031 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.781125 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4tjb4"] Sep 07 11:28:51.794964 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.794935 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5dbqd"] Sep 07 11:28:51.797392 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:51.797358 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-5dbqd"] Sep 07 11:28:52.123734 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:52.123693 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d33a8e0-0074-4449-87e5-397758923bf7" path="/var/lib/kubelet/pods/0d33a8e0-0074-4449-87e5-397758923bf7/volumes" Sep 07 11:28:52.124136 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:52.124115 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb048d7-31f0-40e7-84ec-9bfacf57df36" path="/var/lib/kubelet/pods/ebb048d7-31f0-40e7-84ec-9bfacf57df36/volumes" Sep 07 11:28:55.400999 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:55.400955 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 07 11:28:57.252545 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:28:57.252502 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 07 11:29:05.401263 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:05.401217 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 07 11:29:07.252427 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:07.252382 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 07 11:29:15.401755 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:15.401702 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Sep 07 11:29:15.402165 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:15.401821 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:29:17.252057 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:17.252009 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused" Sep 07 11:29:17.252454 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:17.252130 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:29:19.872160 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:19.872079 2571 generic.go:356] "Generic (PLEG): container finished" podID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerID="642cf588ec6bc9a2e045d34f9f808aefd15aed77ab210c60f55830c6a7a74186" exitCode=137 Sep 07 11:29:19.872621 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:19.872154 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" event={"ID":"561ba036-bb2d-49b5-91e2-0b252e71d3ed","Type":"ContainerDied","Data":"642cf588ec6bc9a2e045d34f9f808aefd15aed77ab210c60f55830c6a7a74186"} Sep 07 11:29:19.874617 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:19.874592 2571 generic.go:356] "Generic (PLEG): container finished" podID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerID="a89bc7c8c445627891d579eaaab2973e486beaead844598e6ce886c77cd7dcf7" exitCode=137 Sep 07 11:29:19.874730 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:19.874634 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" event={"ID":"66645353-b1b3-4b61-953b-bdabd5178dc4","Type":"ContainerDied","Data":"a89bc7c8c445627891d579eaaab2973e486beaead844598e6ce886c77cd7dcf7"} Sep 07 11:29:20.167488 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.167459 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:29:20.195505 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.195481 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:29:20.306237 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.306198 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-online-tls\") pod \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " Sep 07 11:29:20.306237 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.306251 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-registry-tls\") pod \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " Sep 07 11:29:20.306497 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.306293 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssd4p\" (UniqueName: \"kubernetes.io/projected/561ba036-bb2d-49b5-91e2-0b252e71d3ed-kube-api-access-ssd4p\") pod \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " Sep 07 11:29:20.306497 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.306316 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/561ba036-bb2d-49b5-91e2-0b252e71d3ed-feast-data\") pod \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\" (UID: \"561ba036-bb2d-49b5-91e2-0b252e71d3ed\") " Sep 07 11:29:20.306497 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.306350 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/66645353-b1b3-4b61-953b-bdabd5178dc4-feast-data\") pod \"66645353-b1b3-4b61-953b-bdabd5178dc4\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " Sep 07 11:29:20.306497 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.306383 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2m2r\" (UniqueName: \"kubernetes.io/projected/66645353-b1b3-4b61-953b-bdabd5178dc4-kube-api-access-v2m2r\") pod \"66645353-b1b3-4b61-953b-bdabd5178dc4\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " Sep 07 11:29:20.306497 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.306405 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-registry-tls\") pod \"66645353-b1b3-4b61-953b-bdabd5178dc4\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " Sep 07 11:29:20.306497 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.306442 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-online-tls\") pod \"66645353-b1b3-4b61-953b-bdabd5178dc4\" (UID: \"66645353-b1b3-4b61-953b-bdabd5178dc4\") " Sep 07 11:29:20.306992 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.306953 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561ba036-bb2d-49b5-91e2-0b252e71d3ed-feast-data" (OuterVolumeSpecName: "feast-data") pod "561ba036-bb2d-49b5-91e2-0b252e71d3ed" (UID: "561ba036-bb2d-49b5-91e2-0b252e71d3ed"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:29:20.308857 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.308816 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "66645353-b1b3-4b61-953b-bdabd5178dc4" (UID: "66645353-b1b3-4b61-953b-bdabd5178dc4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:29:20.309054 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.309018 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-online-tls" (OuterVolumeSpecName: "online-tls") pod "561ba036-bb2d-49b5-91e2-0b252e71d3ed" (UID: "561ba036-bb2d-49b5-91e2-0b252e71d3ed"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:29:20.309168 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.309058 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561ba036-bb2d-49b5-91e2-0b252e71d3ed-kube-api-access-ssd4p" (OuterVolumeSpecName: "kube-api-access-ssd4p") pod "561ba036-bb2d-49b5-91e2-0b252e71d3ed" (UID: "561ba036-bb2d-49b5-91e2-0b252e71d3ed"). InnerVolumeSpecName "kube-api-access-ssd4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:29:20.309168 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.309101 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "561ba036-bb2d-49b5-91e2-0b252e71d3ed" (UID: "561ba036-bb2d-49b5-91e2-0b252e71d3ed"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:29:20.309168 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.309123 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66645353-b1b3-4b61-953b-bdabd5178dc4-kube-api-access-v2m2r" (OuterVolumeSpecName: "kube-api-access-v2m2r") pod "66645353-b1b3-4b61-953b-bdabd5178dc4" (UID: "66645353-b1b3-4b61-953b-bdabd5178dc4"). InnerVolumeSpecName "kube-api-access-v2m2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:29:20.309289 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.309172 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-online-tls" (OuterVolumeSpecName: "online-tls") pod "66645353-b1b3-4b61-953b-bdabd5178dc4" (UID: "66645353-b1b3-4b61-953b-bdabd5178dc4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:29:20.315802 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.315780 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66645353-b1b3-4b61-953b-bdabd5178dc4-feast-data" (OuterVolumeSpecName: "feast-data") pod "66645353-b1b3-4b61-953b-bdabd5178dc4" (UID: "66645353-b1b3-4b61-953b-bdabd5178dc4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:29:20.407907 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.407806 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:29:20.407907 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.407853 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/561ba036-bb2d-49b5-91e2-0b252e71d3ed-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:29:20.407907 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.407865 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssd4p\" (UniqueName: \"kubernetes.io/projected/561ba036-bb2d-49b5-91e2-0b252e71d3ed-kube-api-access-ssd4p\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:29:20.407907 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.407874 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/561ba036-bb2d-49b5-91e2-0b252e71d3ed-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:29:20.407907 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.407882 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/66645353-b1b3-4b61-953b-bdabd5178dc4-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:29:20.407907 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.407890 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2m2r\" (UniqueName: \"kubernetes.io/projected/66645353-b1b3-4b61-953b-bdabd5178dc4-kube-api-access-v2m2r\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:29:20.407907 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.407897 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:29:20.407907 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.407906 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/66645353-b1b3-4b61-953b-bdabd5178dc4-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:29:20.881958 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.881925 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" Sep 07 11:29:20.882432 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.881930 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt" event={"ID":"66645353-b1b3-4b61-953b-bdabd5178dc4","Type":"ContainerDied","Data":"b5162a49062e6156bd3ab3fdd06ced8994e675981463de33aece664448979fa1"} Sep 07 11:29:20.882432 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.882071 2571 scope.go:117] "RemoveContainer" containerID="d0c65c4407ea720620cad3c2b3140e623a5349175cb2bc04416a971f3cd4f3fe" Sep 07 11:29:20.884482 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.884459 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" event={"ID":"561ba036-bb2d-49b5-91e2-0b252e71d3ed","Type":"ContainerDied","Data":"b31452e74b7af804356d49d9528a55eaa95443b7dae17cd7a1bf48321538aff7"} Sep 07 11:29:20.884630 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.884507 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl" Sep 07 11:29:20.892182 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.892032 2571 scope.go:117] "RemoveContainer" containerID="a89bc7c8c445627891d579eaaab2973e486beaead844598e6ce886c77cd7dcf7" Sep 07 11:29:20.899782 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.899757 2571 scope.go:117] "RemoveContainer" containerID="a76bff70b6c25c1a6d5938c4a6f804b5f33554d66e5675835d74033b4338b6b9" Sep 07 11:29:20.907535 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.907515 2571 scope.go:117] "RemoveContainer" containerID="b7ed943d4a86c28e0d8564e87d6a46e9780b53ee23a52a0cb58384176f7e278b" Sep 07 11:29:20.908459 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.908433 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl"] Sep 07 11:29:20.911943 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.911919 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-677444b594-rdpwl"] Sep 07 11:29:20.916045 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.916026 2571 scope.go:117] "RemoveContainer" containerID="115bc6ddfacef37ef0824588ece426d6090a731be7a855845184e5e56143586f" Sep 07 11:29:20.923282 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.923253 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt"] Sep 07 11:29:20.924093 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.924077 2571 scope.go:117] "RemoveContainer" containerID="642cf588ec6bc9a2e045d34f9f808aefd15aed77ab210c60f55830c6a7a74186" Sep 07 11:29:20.926782 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.926758 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6bd754576f-dhckt"] Sep 07 11:29:20.932339 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.932321 2571 scope.go:117] "RemoveContainer" containerID="18ea8c6fde455952407195e02d48fc99fb7f0caf7b96fe958389d48c888691a8" Sep 07 11:29:20.942115 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:20.942090 2571 scope.go:117] "RemoveContainer" containerID="455a50c4db54ece558d6fa066625d05037ae46889f7e551c58f3edd56c1585fb" Sep 07 11:29:22.123613 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:22.123581 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" path="/var/lib/kubelet/pods/561ba036-bb2d-49b5-91e2-0b252e71d3ed/volumes" Sep 07 11:29:22.124069 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:22.124056 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" path="/var/lib/kubelet/pods/66645353-b1b3-4b61-953b-bdabd5178dc4/volumes" Sep 07 11:29:24.310872 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.310827 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh"] Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311104 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="feast-init" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311115 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="feast-init" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311126 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d33a8e0-0074-4449-87e5-397758923bf7" containerName="redis" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311132 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d33a8e0-0074-4449-87e5-397758923bf7" containerName="redis" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311138 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb048d7-31f0-40e7-84ec-9bfacf57df36" containerName="postgres" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311144 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb048d7-31f0-40e7-84ec-9bfacf57df36" containerName="postgres" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311153 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="online" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311158 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="online" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311164 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085001ba-6664-4434-be0b-fee420861e51" containerName="manager" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311168 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="085001ba-6664-4434-be0b-fee420861e51" containerName="manager" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311173 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="feast-apply" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311178 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="feast-apply" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311187 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="feast-apply" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311191 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="feast-apply" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311197 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="online" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311202 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="online" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311211 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="feast-init" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311216 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="feast-init" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311225 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="registry" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311229 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="registry" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311235 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="registry" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311239 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="registry" Sep 07 11:29:24.311275 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311285 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="085001ba-6664-4434-be0b-fee420861e51" containerName="manager" Sep 07 11:29:24.312187 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311294 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d33a8e0-0074-4449-87e5-397758923bf7" containerName="redis" Sep 07 11:29:24.312187 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311309 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb048d7-31f0-40e7-84ec-9bfacf57df36" containerName="postgres" Sep 07 11:29:24.312187 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311315 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="registry" Sep 07 11:29:24.312187 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311321 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="online" Sep 07 11:29:24.312187 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311327 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="66645353-b1b3-4b61-953b-bdabd5178dc4" containerName="online" Sep 07 11:29:24.312187 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.311332 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="561ba036-bb2d-49b5-91e2-0b252e71d3ed" containerName="registry" Sep 07 11:29:24.316845 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.316821 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.321863 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.321821 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 07 11:29:24.322153 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.321871 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 07 11:29:24.322290 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.322260 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 07 11:29:24.323792 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.323237 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 07 11:29:24.323792 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.323253 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 07 11:29:24.323792 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.323265 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-7v5hj\"" Sep 07 11:29:24.323792 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.323268 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 07 11:29:24.332461 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.332428 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh"] Sep 07 11:29:24.337644 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.337615 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.337792 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.337662 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.337792 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.337722 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/07053a61-efcf-4323-acd1-5af5015c2931-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.337859 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.337791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.337859 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.337816 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gtmt\" (UniqueName: \"kubernetes.io/projected/07053a61-efcf-4323-acd1-5af5015c2931-kube-api-access-2gtmt\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.337859 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.337839 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.438914 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.438873 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/07053a61-efcf-4323-acd1-5af5015c2931-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.439077 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.438930 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.439077 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.438960 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gtmt\" (UniqueName: \"kubernetes.io/projected/07053a61-efcf-4323-acd1-5af5015c2931-kube-api-access-2gtmt\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.439077 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.438996 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.439077 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.439029 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.439077 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.439068 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.439316 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:29:24.439087 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 07 11:29:24.439316 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:29:24.439147 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:29:24.439316 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:29:24.439180 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-offline-tls podName:07053a61-efcf-4323-acd1-5af5015c2931 nodeName:}" failed. No retries permitted until 2026-09-07 11:29:24.939158826 +0000 UTC m=+1737.413515202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-zhpsh" (UID: "07053a61-efcf-4323-acd1-5af5015c2931") : secret "feast-simple-feast-setup-offline-tls" not found Sep 07 11:29:24.439316 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:29:24.439209 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-registry-tls podName:07053a61-efcf-4323-acd1-5af5015c2931 nodeName:}" failed. No retries permitted until 2026-09-07 11:29:24.939199161 +0000 UTC m=+1737.413555537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-zhpsh" (UID: "07053a61-efcf-4323-acd1-5af5015c2931") : secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:29:24.439316 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.439289 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/07053a61-efcf-4323-acd1-5af5015c2931-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.441404 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.441382 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.441507 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.441490 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.448488 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.448458 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gtmt\" (UniqueName: \"kubernetes.io/projected/07053a61-efcf-4323-acd1-5af5015c2931-kube-api-access-2gtmt\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.943576 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.943513 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.943774 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.943627 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.945979 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.945947 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:24.946111 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:24.945996 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-zhpsh\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:25.225811 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:25.225714 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:25.505743 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:25.505592 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh"] Sep 07 11:29:25.507951 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:29:25.507919 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07053a61_efcf_4323_acd1_5af5015c2931.slice/crio-c264ce01919eba91129008196452d94402b4daa73d528a6b06dce31348d4958c WatchSource:0}: Error finding container c264ce01919eba91129008196452d94402b4daa73d528a6b06dce31348d4958c: Status 404 returned error can't find the container with id c264ce01919eba91129008196452d94402b4daa73d528a6b06dce31348d4958c Sep 07 11:29:25.510113 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:25.510097 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 11:29:25.909340 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:25.909306 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerStarted","Data":"c264ce01919eba91129008196452d94402b4daa73d528a6b06dce31348d4958c"} Sep 07 11:29:37.967461 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:37.967424 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerStarted","Data":"2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e"} Sep 07 11:29:42.988616 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:42.988518 2571 generic.go:356] "Generic (PLEG): container finished" podID="07053a61-efcf-4323-acd1-5af5015c2931" containerID="2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e" exitCode=0 Sep 07 11:29:42.988616 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:42.988593 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerDied","Data":"2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e"} Sep 07 11:29:43.994342 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:43.994300 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerStarted","Data":"241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797"} Sep 07 11:29:49.016344 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:49.016313 2571 generic.go:356] "Generic (PLEG): container finished" podID="07053a61-efcf-4323-acd1-5af5015c2931" containerID="241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797" exitCode=0 Sep 07 11:29:49.016754 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:49.016366 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerDied","Data":"241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797"} Sep 07 11:29:50.026464 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:50.026411 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerStarted","Data":"94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2"} Sep 07 11:29:50.027001 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:50.026492 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerStarted","Data":"a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf"} Sep 07 11:29:50.027001 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:50.026514 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerStarted","Data":"99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21"} Sep 07 11:29:50.027001 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:50.026533 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerStarted","Data":"b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277"} Sep 07 11:29:50.054700 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:50.054603 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podStartSLOduration=13.83958532 podStartE2EDuration="26.054580198s" totalImagesPullingTime="12.214994878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:29:50.052632577 +0000 UTC m=+1762.526988946" watchObservedRunningTime="2026-09-07 11:29:50.054580198 +0000 UTC m=+1762.528936579" Sep 07 11:29:52.226019 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:52.225972 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:52.226766 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:52.226734 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:52.227682 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:52.227661 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:52.227822 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:52.227690 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:52.228084 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:52.228056 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.32:6571: connect: connection refused" Sep 07 11:29:52.228214 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:52.228104 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Sep 07 11:29:52.228363 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:52.228342 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.32:8443: connect: connection refused" Sep 07 11:29:52.228633 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:52.228173 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.32:8816: connect: connection refused" Sep 07 11:29:55.226800 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:55.226748 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:55.227322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:55.226821 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:55.227322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:55.227002 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.32:8816: connect: connection refused" Sep 07 11:29:55.227322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:55.227001 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.32:6571: connect: connection refused" Sep 07 11:29:55.227512 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:55.227377 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.32:8443: connect: connection refused" Sep 07 11:29:55.227512 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:55.227489 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:55.227653 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:55.227528 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:55.227653 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:55.227588 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Sep 07 11:29:58.227267 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:58.227233 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:58.227790 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:58.227361 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:58.227790 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:58.227634 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:58.228244 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:58.228216 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:58.228392 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:58.228373 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:58.228525 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:58.228507 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:58.232698 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:58.232678 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:29:59.074775 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:29:59.074731 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:30:28.082619 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:30:28.082534 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:30:28.085166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:30:28.085132 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:31:01.611450 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:01.611404 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh"] Sep 07 11:31:01.611982 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:01.611695 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="registry" containerID="cri-o://b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277" gracePeriod=30 Sep 07 11:31:01.611982 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:01.611741 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="ui" containerID="cri-o://94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2" gracePeriod=30 Sep 07 11:31:01.611982 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:01.611758 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="offline" containerID="cri-o://a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf" gracePeriod=30 Sep 07 11:31:01.611982 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:01.611758 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="online" containerID="cri-o://99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21" gracePeriod=30 Sep 07 11:31:02.026663 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.026608 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw"] Sep 07 11:31:02.031671 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.031641 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.034249 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.034225 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-hwlsg\"" Sep 07 11:31:02.042376 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.042351 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw"] Sep 07 11:31:02.181600 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.181560 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.181807 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.181612 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/533a46d7-e4eb-4b1b-bbed-4271bf401e68-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.181807 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.181641 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.181807 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.181717 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.181807 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.181779 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2zh\" (UniqueName: \"kubernetes.io/projected/533a46d7-e4eb-4b1b-bbed-4271bf401e68-kube-api-access-jn2zh\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.181807 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.181807 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.283036 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.282941 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.283036 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.282984 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.283036 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.283019 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2zh\" (UniqueName: \"kubernetes.io/projected/533a46d7-e4eb-4b1b-bbed-4271bf401e68-kube-api-access-jn2zh\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.283352 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.283046 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.283352 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.283104 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.283352 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.283144 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/533a46d7-e4eb-4b1b-bbed-4271bf401e68-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.283352 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:31:02.283156 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:31:02.283352 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:31:02.283222 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-registry-tls podName:533a46d7-e4eb-4b1b-bbed-4271bf401e68 nodeName:}" failed. No retries permitted until 2026-09-07 11:31:02.783203825 +0000 UTC m=+1835.257560194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-4f5gw" (UID: "533a46d7-e4eb-4b1b-bbed-4271bf401e68") : secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:31:02.283657 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.283533 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/533a46d7-e4eb-4b1b-bbed-4271bf401e68-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.285884 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.285846 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.286024 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.285896 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.286024 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.285853 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.296042 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.296004 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2zh\" (UniqueName: \"kubernetes.io/projected/533a46d7-e4eb-4b1b-bbed-4271bf401e68-kube-api-access-jn2zh\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.787289 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.787235 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.790934 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.790897 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4f5gw\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:02.941273 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:02.941222 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:03.236237 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:03.236201 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw"] Sep 07 11:31:03.238510 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:31:03.238457 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod533a46d7_e4eb_4b1b_bbed_4271bf401e68.slice/crio-1b66b61b5856c1f4a8054eea03a2091dbfbc68150417a571d443e819d139c680 WatchSource:0}: Error finding container 1b66b61b5856c1f4a8054eea03a2091dbfbc68150417a571d443e819d139c680: Status 404 returned error can't find the container with id 1b66b61b5856c1f4a8054eea03a2091dbfbc68150417a571d443e819d139c680 Sep 07 11:31:03.324097 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:03.324063 2571 generic.go:356] "Generic (PLEG): container finished" podID="07053a61-efcf-4323-acd1-5af5015c2931" containerID="94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2" exitCode=0 Sep 07 11:31:03.324097 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:03.324091 2571 generic.go:356] "Generic (PLEG): container finished" podID="07053a61-efcf-4323-acd1-5af5015c2931" containerID="99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21" exitCode=0 Sep 07 11:31:03.324323 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:03.324136 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerDied","Data":"94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2"} Sep 07 11:31:03.324323 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:03.324185 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerDied","Data":"99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21"} Sep 07 11:31:03.325674 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:03.325646 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerStarted","Data":"241e3b76c5f51cfde54e528126765832ca6e5cdb6c7d5cf634e4ab0d726dd02a"} Sep 07 11:31:03.325794 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:03.325687 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerStarted","Data":"1b66b61b5856c1f4a8054eea03a2091dbfbc68150417a571d443e819d139c680"} Sep 07 11:31:08.228790 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:08.228738 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.32:8443: connect: connection refused" Sep 07 11:31:09.071508 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:09.071450 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Sep 07 11:31:09.352179 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:09.352087 2571 generic.go:356] "Generic (PLEG): container finished" podID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerID="241e3b76c5f51cfde54e528126765832ca6e5cdb6c7d5cf634e4ab0d726dd02a" exitCode=0 Sep 07 11:31:09.352657 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:09.352164 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerDied","Data":"241e3b76c5f51cfde54e528126765832ca6e5cdb6c7d5cf634e4ab0d726dd02a"} Sep 07 11:31:10.358120 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:10.358081 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerStarted","Data":"295632a47579ace8e681e0ea56275b7519fdd315afd241f74803d7d0d108c788"} Sep 07 11:31:16.384013 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:16.383977 2571 generic.go:356] "Generic (PLEG): container finished" podID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerID="295632a47579ace8e681e0ea56275b7519fdd315afd241f74803d7d0d108c788" exitCode=0 Sep 07 11:31:16.384455 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:16.384063 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerDied","Data":"295632a47579ace8e681e0ea56275b7519fdd315afd241f74803d7d0d108c788"} Sep 07 11:31:17.394029 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.393981 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerStarted","Data":"302163644f84f9c7332c1a6af9a2a886c7ff7298be3a92e01d48ebfec296eba1"} Sep 07 11:31:17.394518 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.394051 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerStarted","Data":"13b8301dcf98a9b7e35273d34dab32f55c54f9defc70927805f7bf5d45e7a75f"} Sep 07 11:31:17.394518 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.394075 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerStarted","Data":"44bd7999a25a6536cfbd367e57c0d9d82268f6722826e14194f48917ce97be94"} Sep 07 11:31:17.394518 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.394092 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerStarted","Data":"fb1faed985d05f42e43d23167fee1f48b6eaa8c05fa3d37459fbc9bec9ca2ba1"} Sep 07 11:31:17.942311 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.942165 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:17.942311 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.942233 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:17.942311 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.942255 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:17.942311 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.942273 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:17.944183 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.944135 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Sep 07 11:31:17.944183 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.944158 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.33:8443: connect: connection refused" Sep 07 11:31:17.944390 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.944154 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.33:8816: connect: connection refused" Sep 07 11:31:17.944390 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:17.944259 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Sep 07 11:31:18.229139 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:18.229009 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.32:8443: connect: connection refused" Sep 07 11:31:19.071774 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:19.071720 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Sep 07 11:31:20.941839 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:20.941787 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.33:8443: connect: connection refused" Sep 07 11:31:20.942427 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:20.941807 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Sep 07 11:31:20.942697 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:20.941815 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Sep 07 11:31:20.942780 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:20.941870 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.33:8816: connect: connection refused" Sep 07 11:31:22.941747 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:22.941693 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:22.942269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:22.941777 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:22.942269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:22.941802 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:22.942269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:22.941822 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:23.942435 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:23.942382 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Sep 07 11:31:23.943009 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:23.942394 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.33:8443: connect: connection refused" Sep 07 11:31:23.943009 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:23.942410 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Sep 07 11:31:23.943163 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:23.943118 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.33:8816: connect: connection refused" Sep 07 11:31:26.942790 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:26.942760 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:26.943204 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:26.942872 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:26.943204 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:26.943128 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:26.943722 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:26.943696 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:26.943831 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:26.943795 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:26.943869 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:26.943839 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:26.947843 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:26.947805 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:26.969522 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:26.969458 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podStartSLOduration=24.969417619 podStartE2EDuration="24.969417619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:31:17.42050737 +0000 UTC m=+1849.894863748" watchObservedRunningTime="2026-09-07 11:31:26.969417619 +0000 UTC m=+1859.443773998" Sep 07 11:31:27.452967 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:27.452907 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:31:28.228293 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:28.228225 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.32:8443: connect: connection refused" Sep 07 11:31:28.228856 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:28.228424 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:31:29.071974 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:29.071923 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Sep 07 11:31:29.072169 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:29.072049 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:31:32.279869 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.279843 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:31:32.363709 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.363675 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-ui-tls\") pod \"07053a61-efcf-4323-acd1-5af5015c2931\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " Sep 07 11:31:32.363709 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.363713 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-registry-tls\") pod \"07053a61-efcf-4323-acd1-5af5015c2931\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " Sep 07 11:31:32.363970 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.363738 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-offline-tls\") pod \"07053a61-efcf-4323-acd1-5af5015c2931\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " Sep 07 11:31:32.363970 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.363778 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/07053a61-efcf-4323-acd1-5af5015c2931-feast-data\") pod \"07053a61-efcf-4323-acd1-5af5015c2931\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " Sep 07 11:31:32.363970 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.363864 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gtmt\" (UniqueName: \"kubernetes.io/projected/07053a61-efcf-4323-acd1-5af5015c2931-kube-api-access-2gtmt\") pod \"07053a61-efcf-4323-acd1-5af5015c2931\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " Sep 07 11:31:32.363970 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.363896 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-online-tls\") pod \"07053a61-efcf-4323-acd1-5af5015c2931\" (UID: \"07053a61-efcf-4323-acd1-5af5015c2931\") " Sep 07 11:31:32.364436 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.364400 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07053a61-efcf-4323-acd1-5af5015c2931-feast-data" (OuterVolumeSpecName: "feast-data") pod "07053a61-efcf-4323-acd1-5af5015c2931" (UID: "07053a61-efcf-4323-acd1-5af5015c2931"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:31:32.366126 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.366089 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "07053a61-efcf-4323-acd1-5af5015c2931" (UID: "07053a61-efcf-4323-acd1-5af5015c2931"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:31:32.366126 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.366106 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07053a61-efcf-4323-acd1-5af5015c2931-kube-api-access-2gtmt" (OuterVolumeSpecName: "kube-api-access-2gtmt") pod "07053a61-efcf-4323-acd1-5af5015c2931" (UID: "07053a61-efcf-4323-acd1-5af5015c2931"). InnerVolumeSpecName "kube-api-access-2gtmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:31:32.366298 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.366137 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-online-tls" (OuterVolumeSpecName: "online-tls") pod "07053a61-efcf-4323-acd1-5af5015c2931" (UID: "07053a61-efcf-4323-acd1-5af5015c2931"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:31:32.366298 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.366266 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "07053a61-efcf-4323-acd1-5af5015c2931" (UID: "07053a61-efcf-4323-acd1-5af5015c2931"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:31:32.366405 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.366313 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "07053a61-efcf-4323-acd1-5af5015c2931" (UID: "07053a61-efcf-4323-acd1-5af5015c2931"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:31:32.464481 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.464400 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gtmt\" (UniqueName: \"kubernetes.io/projected/07053a61-efcf-4323-acd1-5af5015c2931-kube-api-access-2gtmt\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:31:32.464481 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.464427 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:31:32.464481 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.464438 2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-ui-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:31:32.464481 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.464447 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:31:32.464481 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.464454 2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/07053a61-efcf-4323-acd1-5af5015c2931-offline-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:31:32.464481 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.464461 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/07053a61-efcf-4323-acd1-5af5015c2931-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:31:32.477695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.477660 2571 generic.go:356] "Generic (PLEG): container finished" podID="07053a61-efcf-4323-acd1-5af5015c2931" containerID="a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf" exitCode=137 Sep 07 11:31:32.477695 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.477688 2571 generic.go:356] "Generic (PLEG): container finished" podID="07053a61-efcf-4323-acd1-5af5015c2931" containerID="b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277" exitCode=137 Sep 07 11:31:32.477883 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.477731 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" Sep 07 11:31:32.477883 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.477737 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerDied","Data":"a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf"} Sep 07 11:31:32.477883 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.477788 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerDied","Data":"b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277"} Sep 07 11:31:32.477883 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.477804 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh" event={"ID":"07053a61-efcf-4323-acd1-5af5015c2931","Type":"ContainerDied","Data":"c264ce01919eba91129008196452d94402b4daa73d528a6b06dce31348d4958c"} Sep 07 11:31:32.477883 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.477821 2571 scope.go:117] "RemoveContainer" containerID="94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2" Sep 07 11:31:32.486290 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.486272 2571 scope.go:117] "RemoveContainer" containerID="a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf" Sep 07 11:31:32.493984 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.493967 2571 scope.go:117] "RemoveContainer" containerID="99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21" Sep 07 11:31:32.501833 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.501806 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh"] Sep 07 11:31:32.502112 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.502097 2571 scope.go:117] "RemoveContainer" containerID="b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277" Sep 07 11:31:32.504646 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.504614 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-zhpsh"] Sep 07 11:31:32.509785 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.509764 2571 scope.go:117] "RemoveContainer" containerID="241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797" Sep 07 11:31:32.516515 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.516498 2571 scope.go:117] "RemoveContainer" containerID="2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e" Sep 07 11:31:32.523455 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.523437 2571 scope.go:117] "RemoveContainer" containerID="94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2" Sep 07 11:31:32.523733 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:31:32.523717 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2\": container with ID starting with 94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2 not found: ID does not exist" containerID="94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2" Sep 07 11:31:32.523791 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.523742 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2"} err="failed to get container status \"94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2\": rpc error: code = NotFound desc = could not find container \"94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2\": container with ID starting with 94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2 not found: ID does not exist" Sep 07 11:31:32.523791 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.523759 2571 scope.go:117] "RemoveContainer" containerID="a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf" Sep 07 11:31:32.524000 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:31:32.523980 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf\": container with ID starting with a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf not found: ID does not exist" containerID="a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf" Sep 07 11:31:32.524057 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.524009 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf"} err="failed to get container status \"a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf\": rpc error: code = NotFound desc = could not find container \"a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf\": container with ID starting with a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf not found: ID does not exist" Sep 07 11:31:32.524057 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.524031 2571 scope.go:117] "RemoveContainer" containerID="99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21" Sep 07 11:31:32.524256 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:31:32.524241 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21\": container with ID starting with 99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21 not found: ID does not exist" containerID="99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21" Sep 07 11:31:32.524294 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.524260 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21"} err="failed to get container status \"99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21\": rpc error: code = NotFound desc = could not find container \"99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21\": container with ID starting with 99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21 not found: ID does not exist" Sep 07 11:31:32.524294 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.524272 2571 scope.go:117] "RemoveContainer" containerID="b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277" Sep 07 11:31:32.524463 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:31:32.524448 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277\": container with ID starting with b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277 not found: ID does not exist" containerID="b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277" Sep 07 11:31:32.524507 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.524465 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277"} err="failed to get container status \"b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277\": rpc error: code = NotFound desc = could not find container \"b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277\": container with ID starting with b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277 not found: ID does not exist" Sep 07 11:31:32.524507 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.524478 2571 scope.go:117] "RemoveContainer" containerID="241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797" Sep 07 11:31:32.524754 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:31:32.524739 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797\": container with ID starting with 241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797 not found: ID does not exist" containerID="241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797" Sep 07 11:31:32.524799 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.524760 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797"} err="failed to get container status \"241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797\": rpc error: code = NotFound desc = could not find container \"241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797\": container with ID starting with 241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797 not found: ID does not exist" Sep 07 11:31:32.524799 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.524772 2571 scope.go:117] "RemoveContainer" containerID="2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e" Sep 07 11:31:32.525057 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:31:32.525041 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e\": container with ID starting with 2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e not found: ID does not exist" containerID="2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e" Sep 07 11:31:32.525119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.525073 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e"} err="failed to get container status \"2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e\": rpc error: code = NotFound desc = could not find container \"2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e\": container with ID starting with 2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e not found: ID does not exist" Sep 07 11:31:32.525119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.525094 2571 scope.go:117] "RemoveContainer" containerID="94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2" Sep 07 11:31:32.525335 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.525317 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2"} err="failed to get container status \"94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2\": rpc error: code = NotFound desc = could not find container \"94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2\": container with ID starting with 94dd63702da527050ce386c6eb44c64219009ed3ec82301a01c9d0632c8d94f2 not found: ID does not exist" Sep 07 11:31:32.525384 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.525337 2571 scope.go:117] "RemoveContainer" containerID="a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf" Sep 07 11:31:32.525541 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.525524 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf"} err="failed to get container status \"a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf\": rpc error: code = NotFound desc = could not find container \"a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf\": container with ID starting with a62b6c9318c684cedd10e1ac5bed070d47af948793542a42d42cd2f08ff2acbf not found: ID does not exist" Sep 07 11:31:32.525637 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.525542 2571 scope.go:117] "RemoveContainer" containerID="99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21" Sep 07 11:31:32.525811 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.525794 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21"} err="failed to get container status \"99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21\": rpc error: code = NotFound desc = could not find container \"99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21\": container with ID starting with 99fdefc335323238207d078201e0d710ccbf4804c31f9da11dc05aa910c22b21 not found: ID does not exist" Sep 07 11:31:32.525857 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.525812 2571 scope.go:117] "RemoveContainer" containerID="b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277" Sep 07 11:31:32.526019 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.526002 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277"} err="failed to get container status \"b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277\": rpc error: code = NotFound desc = could not find container \"b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277\": container with ID starting with b971cfae8928c3407791e9d9105bc07ed77caa9e90d7aa9b29105287472ff277 not found: ID does not exist" Sep 07 11:31:32.526084 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.526020 2571 scope.go:117] "RemoveContainer" containerID="241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797" Sep 07 11:31:32.526239 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.526222 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797"} err="failed to get container status \"241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797\": rpc error: code = NotFound desc = could not find container \"241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797\": container with ID starting with 241ea5a728c51e4cf845c001b2f9f60b4845c4eb5605a0d3793fe38558acd797 not found: ID does not exist" Sep 07 11:31:32.526284 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.526240 2571 scope.go:117] "RemoveContainer" containerID="2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e" Sep 07 11:31:32.526446 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:32.526429 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e"} err="failed to get container status \"2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e\": rpc error: code = NotFound desc = could not find container \"2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e\": container with ID starting with 2572540c06f04394c9b48f98a696b32ddadc3ebdb9792c9b9a643275661a7e2e not found: ID does not exist" Sep 07 11:31:34.123480 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:31:34.123437 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07053a61-efcf-4323-acd1-5af5015c2931" path="/var/lib/kubelet/pods/07053a61-efcf-4323-acd1-5af5015c2931/volumes" Sep 07 11:32:29.252432 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252377 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7"] Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252713 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="ui" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252728 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="ui" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252739 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="registry" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252745 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="registry" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252803 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="feast-init" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252814 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="feast-init" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252825 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="offline" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252831 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="offline" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252855 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="feast-apply" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252864 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="feast-apply" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252873 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="online" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252881 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="online" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252960 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="registry" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252973 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="offline" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252979 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="ui" Sep 07 11:32:29.253030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.252986 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="07053a61-efcf-4323-acd1-5af5015c2931" containerName="online" Sep 07 11:32:29.257727 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.257709 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.260597 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.260571 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 07 11:32:29.260757 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.260641 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 07 11:32:29.260757 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.260718 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 07 11:32:29.260895 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.260781 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-g2rtb\"" Sep 07 11:32:29.261004 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.260988 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 07 11:32:29.261768 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.261747 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 07 11:32:29.261902 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.261773 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 07 11:32:29.278179 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.278147 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7"] Sep 07 11:32:29.343195 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.343146 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.343195 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.343189 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.343441 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.343226 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75ft\" (UniqueName: \"kubernetes.io/projected/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-kube-api-access-x75ft\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.343441 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.343247 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.343441 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.343266 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.343441 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.343280 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.443861 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.443803 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.443861 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.443854 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.444188 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.443888 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75ft\" (UniqueName: \"kubernetes.io/projected/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-kube-api-access-x75ft\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.444188 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.443910 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.444188 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.443934 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.444188 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.443954 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.444188 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:32:29.444104 2571 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 07 11:32:29.444423 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:32:29.444197 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-ui-tls podName:2f88cef7-6702-4aa7-a455-afa2bc59e1c8 nodeName:}" failed. No retries permitted until 2026-09-07 11:32:29.944171958 +0000 UTC m=+1922.418528329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-brms7" (UID: "2f88cef7-6702-4aa7-a455-afa2bc59e1c8") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 07 11:32:29.444491 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.444471 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.444731 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.444703 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.446826 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.446799 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.447004 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.446837 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.454291 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.454250 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75ft\" (UniqueName: \"kubernetes.io/projected/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-kube-api-access-x75ft\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.948436 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.948379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:29.950819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:29.950796 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-brms7\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:30.167319 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:30.167262 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:32:30.445524 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:30.445445 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7"] Sep 07 11:32:30.448136 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:32:30.448105 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f88cef7_6702_4aa7_a455_afa2bc59e1c8.slice/crio-29833c0625223fdba9afff16704efeb459903c5db3237aa08a4936baff7f32d2 WatchSource:0}: Error finding container 29833c0625223fdba9afff16704efeb459903c5db3237aa08a4936baff7f32d2: Status 404 returned error can't find the container with id 29833c0625223fdba9afff16704efeb459903c5db3237aa08a4936baff7f32d2 Sep 07 11:32:30.723517 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:30.723421 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerStarted","Data":"f4620386f133a92cfb9a04f082f74ddbcd151397f123b83ed59185617e5b863f"} Sep 07 11:32:30.723517 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:30.723474 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerStarted","Data":"29833c0625223fdba9afff16704efeb459903c5db3237aa08a4936baff7f32d2"} Sep 07 11:32:35.745778 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:35.745740 2571 generic.go:356] "Generic (PLEG): container finished" podID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerID="f4620386f133a92cfb9a04f082f74ddbcd151397f123b83ed59185617e5b863f" exitCode=0 Sep 07 11:32:35.746183 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:35.745813 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerDied","Data":"f4620386f133a92cfb9a04f082f74ddbcd151397f123b83ed59185617e5b863f"} Sep 07 11:32:36.752849 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:32:36.752809 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerStarted","Data":"76737fdc63280ad1db8e0bf2352e75a437b7299a2e4089f7e023356ac34b45fa"} Sep 07 11:33:01.859041 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:01.859003 2571 generic.go:356] "Generic (PLEG): container finished" podID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerID="76737fdc63280ad1db8e0bf2352e75a437b7299a2e4089f7e023356ac34b45fa" exitCode=0 Sep 07 11:33:01.859475 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:01.859064 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerDied","Data":"76737fdc63280ad1db8e0bf2352e75a437b7299a2e4089f7e023356ac34b45fa"} Sep 07 11:33:02.867432 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:02.867396 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerStarted","Data":"a510a7911d069da22060db0d96bd234a1915f2f356ed33f1421b77b6b0c14d72"} Sep 07 11:33:02.867971 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:02.867445 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerStarted","Data":"06e595780d46c5f37f588bd33daa2c59954e4b991c0b33ac2c5c5a588dbe5ac6"} Sep 07 11:33:02.867971 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:02.867457 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerStarted","Data":"3efcc2c8eb151c2493eed734bd06b16c5bee3462ba81ea48dfbc8a249f997123"} Sep 07 11:33:02.898742 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:02.898671 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podStartSLOduration=33.898620395 podStartE2EDuration="33.898620395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:33:02.896351888 +0000 UTC m=+1955.370708282" watchObservedRunningTime="2026-09-07 11:33:02.898620395 +0000 UTC m=+1955.372976773" Sep 07 11:33:03.168410 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:03.168301 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:33:03.168410 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:03.168382 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:33:03.168410 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:03.168405 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:33:03.169938 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:03.169898 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.34:8443: connect: connection refused" Sep 07 11:33:03.169938 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:03.169923 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.34:8816: connect: connection refused" Sep 07 11:33:03.170141 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:03.169964 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 07 11:33:06.167856 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:06.167808 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.34:8816: connect: connection refused" Sep 07 11:33:06.168252 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:06.167810 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.34:8443: connect: connection refused" Sep 07 11:33:06.168252 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:06.167826 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 07 11:33:09.168563 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:09.168498 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.34:8443: connect: connection refused" Sep 07 11:33:09.169028 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:09.168498 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.34:8816: connect: connection refused" Sep 07 11:33:09.169028 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:09.168516 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 07 11:33:10.168074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:10.168035 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:33:10.168303 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:10.168098 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:33:10.168303 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:10.168115 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:33:12.168642 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:12.168607 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:33:12.169081 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:12.168732 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:33:12.169317 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:12.169288 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:33:12.169641 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:12.169620 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:33:13.106341 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:13.106307 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:33:14.625837 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:33:14.625793 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:34:15.971150 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:15.971116 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7"] Sep 07 11:34:15.971699 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:15.971387 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="online" containerID="cri-o://3efcc2c8eb151c2493eed734bd06b16c5bee3462ba81ea48dfbc8a249f997123" gracePeriod=30 Sep 07 11:34:15.971699 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:15.971437 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="offline" containerID="cri-o://06e595780d46c5f37f588bd33daa2c59954e4b991c0b33ac2c5c5a588dbe5ac6" gracePeriod=30 Sep 07 11:34:15.971699 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:15.971450 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="ui" containerID="cri-o://a510a7911d069da22060db0d96bd234a1915f2f356ed33f1421b77b6b0c14d72" gracePeriod=30 Sep 07 11:34:16.144186 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:16.144149 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw"] Sep 07 11:34:16.144424 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:16.144381 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="registry" containerID="cri-o://fb1faed985d05f42e43d23167fee1f48b6eaa8c05fa3d37459fbc9bec9ca2ba1" gracePeriod=30 Sep 07 11:34:16.144630 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:16.144494 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="offline" containerID="cri-o://13b8301dcf98a9b7e35273d34dab32f55c54f9defc70927805f7bf5d45e7a75f" gracePeriod=30 Sep 07 11:34:16.144630 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:16.144499 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="ui" containerID="cri-o://302163644f84f9c7332c1a6af9a2a886c7ff7298be3a92e01d48ebfec296eba1" gracePeriod=30 Sep 07 11:34:16.144856 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:16.144500 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="online" containerID="cri-o://44bd7999a25a6536cfbd367e57c0d9d82268f6722826e14194f48917ce97be94" gracePeriod=30 Sep 07 11:34:16.943353 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:16.943308 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.33:8443: connect: connection refused" Sep 07 11:34:17.168357 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:17.168262 2571 generic.go:356] "Generic (PLEG): container finished" podID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerID="a510a7911d069da22060db0d96bd234a1915f2f356ed33f1421b77b6b0c14d72" exitCode=0 Sep 07 11:34:17.168357 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:17.168294 2571 generic.go:356] "Generic (PLEG): container finished" podID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerID="3efcc2c8eb151c2493eed734bd06b16c5bee3462ba81ea48dfbc8a249f997123" exitCode=0 Sep 07 11:34:17.168357 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:17.168333 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerDied","Data":"a510a7911d069da22060db0d96bd234a1915f2f356ed33f1421b77b6b0c14d72"} Sep 07 11:34:17.168864 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:17.168381 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerDied","Data":"3efcc2c8eb151c2493eed734bd06b16c5bee3462ba81ea48dfbc8a249f997123"} Sep 07 11:34:17.171147 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:17.171125 2571 generic.go:356] "Generic (PLEG): container finished" podID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerID="302163644f84f9c7332c1a6af9a2a886c7ff7298be3a92e01d48ebfec296eba1" exitCode=0 Sep 07 11:34:17.171147 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:17.171144 2571 generic.go:356] "Generic (PLEG): container finished" podID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerID="44bd7999a25a6536cfbd367e57c0d9d82268f6722826e14194f48917ce97be94" exitCode=0 Sep 07 11:34:17.171298 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:17.171194 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerDied","Data":"302163644f84f9c7332c1a6af9a2a886c7ff7298be3a92e01d48ebfec296eba1"} Sep 07 11:34:17.171298 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:17.171227 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerDied","Data":"44bd7999a25a6536cfbd367e57c0d9d82268f6722826e14194f48917ce97be94"} Sep 07 11:34:17.449814 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:17.449724 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Sep 07 11:34:22.169677 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:22.169629 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.34:8443: connect: connection refused" Sep 07 11:34:23.919750 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:23.919711 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 07 11:34:26.943705 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:26.943652 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.33:8443: connect: connection refused" Sep 07 11:34:27.450048 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:27.450004 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Sep 07 11:34:32.169900 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:32.169850 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.34:8443: connect: connection refused" Sep 07 11:34:33.919939 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:33.919882 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 07 11:34:36.943500 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:36.943454 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.33:8443: connect: connection refused" Sep 07 11:34:36.943980 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:36.943625 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:34:37.449458 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:37.449413 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Sep 07 11:34:37.449665 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:37.449566 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:34:42.169624 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:42.169576 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.34:8443: connect: connection refused" Sep 07 11:34:42.170035 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:42.169712 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:34:43.919495 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:43.919454 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" Sep 07 11:34:43.919933 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:43.919595 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:34:46.289326 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.289293 2571 generic.go:356] "Generic (PLEG): container finished" podID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerID="13b8301dcf98a9b7e35273d34dab32f55c54f9defc70927805f7bf5d45e7a75f" exitCode=137 Sep 07 11:34:46.289326 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.289319 2571 generic.go:356] "Generic (PLEG): container finished" podID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerID="fb1faed985d05f42e43d23167fee1f48b6eaa8c05fa3d37459fbc9bec9ca2ba1" exitCode=137 Sep 07 11:34:46.289788 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.289358 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerDied","Data":"13b8301dcf98a9b7e35273d34dab32f55c54f9defc70927805f7bf5d45e7a75f"} Sep 07 11:34:46.289788 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.289402 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerDied","Data":"fb1faed985d05f42e43d23167fee1f48b6eaa8c05fa3d37459fbc9bec9ca2ba1"} Sep 07 11:34:46.291976 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.291953 2571 generic.go:356] "Generic (PLEG): container finished" podID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerID="06e595780d46c5f37f588bd33daa2c59954e4b991c0b33ac2c5c5a588dbe5ac6" exitCode=137 Sep 07 11:34:46.292079 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.291999 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerDied","Data":"06e595780d46c5f37f588bd33daa2c59954e4b991c0b33ac2c5c5a588dbe5ac6"} Sep 07 11:34:46.609453 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.609430 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:34:46.761964 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.761934 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-offline-tls\") pod \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " Sep 07 11:34:46.762140 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.761976 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-ui-tls\") pod \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " Sep 07 11:34:46.762140 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.762030 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-online-tls\") pod \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " Sep 07 11:34:46.762140 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.762070 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-registry-tls\") pod \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " Sep 07 11:34:46.762140 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.762097 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-feast-data\") pod \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " Sep 07 11:34:46.762341 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.762167 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x75ft\" (UniqueName: \"kubernetes.io/projected/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-kube-api-access-x75ft\") pod \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\" (UID: \"2f88cef7-6702-4aa7-a455-afa2bc59e1c8\") " Sep 07 11:34:46.762651 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.762605 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2f88cef7-6702-4aa7-a455-afa2bc59e1c8" (UID: "2f88cef7-6702-4aa7-a455-afa2bc59e1c8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 11:34:46.762785 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.762758 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-feast-data" (OuterVolumeSpecName: "feast-data") pod "2f88cef7-6702-4aa7-a455-afa2bc59e1c8" (UID: "2f88cef7-6702-4aa7-a455-afa2bc59e1c8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:34:46.764764 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.764738 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-online-tls" (OuterVolumeSpecName: "online-tls") pod "2f88cef7-6702-4aa7-a455-afa2bc59e1c8" (UID: "2f88cef7-6702-4aa7-a455-afa2bc59e1c8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:34:46.764852 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.764772 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2f88cef7-6702-4aa7-a455-afa2bc59e1c8" (UID: "2f88cef7-6702-4aa7-a455-afa2bc59e1c8"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:34:46.764852 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.764808 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2f88cef7-6702-4aa7-a455-afa2bc59e1c8" (UID: "2f88cef7-6702-4aa7-a455-afa2bc59e1c8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:34:46.764939 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.764852 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-kube-api-access-x75ft" (OuterVolumeSpecName: "kube-api-access-x75ft") pod "2f88cef7-6702-4aa7-a455-afa2bc59e1c8" (UID: "2f88cef7-6702-4aa7-a455-afa2bc59e1c8"). InnerVolumeSpecName "kube-api-access-x75ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:34:46.778819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.778799 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:34:46.863693 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.863600 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/533a46d7-e4eb-4b1b-bbed-4271bf401e68-feast-data\") pod \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " Sep 07 11:34:46.863693 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.863646 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-offline-tls\") pod \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " Sep 07 11:34:46.863693 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.863678 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-ui-tls\") pod \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " Sep 07 11:34:46.863972 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.863741 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-online-tls\") pod \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " Sep 07 11:34:46.863972 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.863768 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn2zh\" (UniqueName: \"kubernetes.io/projected/533a46d7-e4eb-4b1b-bbed-4271bf401e68-kube-api-access-jn2zh\") pod \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " Sep 07 11:34:46.863972 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.863813 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-registry-tls\") pod \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\" (UID: \"533a46d7-e4eb-4b1b-bbed-4271bf401e68\") " Sep 07 11:34:46.863972 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.863959 2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-ui-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:46.864186 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.863975 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:46.864186 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.863988 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:46.864186 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.864000 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:46.864186 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.864014 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x75ft\" (UniqueName: \"kubernetes.io/projected/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-kube-api-access-x75ft\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:46.864186 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.864026 2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2f88cef7-6702-4aa7-a455-afa2bc59e1c8-offline-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:46.864433 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.864188 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533a46d7-e4eb-4b1b-bbed-4271bf401e68-feast-data" (OuterVolumeSpecName: "feast-data") pod "533a46d7-e4eb-4b1b-bbed-4271bf401e68" (UID: "533a46d7-e4eb-4b1b-bbed-4271bf401e68"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:34:46.865883 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.865846 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533a46d7-e4eb-4b1b-bbed-4271bf401e68-kube-api-access-jn2zh" (OuterVolumeSpecName: "kube-api-access-jn2zh") pod "533a46d7-e4eb-4b1b-bbed-4271bf401e68" (UID: "533a46d7-e4eb-4b1b-bbed-4271bf401e68"). InnerVolumeSpecName "kube-api-access-jn2zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:34:46.865883 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.865850 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "533a46d7-e4eb-4b1b-bbed-4271bf401e68" (UID: "533a46d7-e4eb-4b1b-bbed-4271bf401e68"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:34:46.866041 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.865914 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-online-tls" (OuterVolumeSpecName: "online-tls") pod "533a46d7-e4eb-4b1b-bbed-4271bf401e68" (UID: "533a46d7-e4eb-4b1b-bbed-4271bf401e68"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:34:46.866078 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.866055 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "533a46d7-e4eb-4b1b-bbed-4271bf401e68" (UID: "533a46d7-e4eb-4b1b-bbed-4271bf401e68"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:34:46.866160 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.866142 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "533a46d7-e4eb-4b1b-bbed-4271bf401e68" (UID: "533a46d7-e4eb-4b1b-bbed-4271bf401e68"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:34:46.965217 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.965179 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:46.965217 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.965210 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2zh\" (UniqueName: \"kubernetes.io/projected/533a46d7-e4eb-4b1b-bbed-4271bf401e68-kube-api-access-jn2zh\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:46.965217 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.965222 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:46.965441 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.965230 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/533a46d7-e4eb-4b1b-bbed-4271bf401e68-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:46.965441 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.965236 2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-offline-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:46.965441 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:46.965244 2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/533a46d7-e4eb-4b1b-bbed-4271bf401e68-ui-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:34:47.298269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.298244 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" Sep 07 11:34:47.298704 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.298273 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw" event={"ID":"533a46d7-e4eb-4b1b-bbed-4271bf401e68","Type":"ContainerDied","Data":"1b66b61b5856c1f4a8054eea03a2091dbfbc68150417a571d443e819d139c680"} Sep 07 11:34:47.298704 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.298336 2571 scope.go:117] "RemoveContainer" containerID="302163644f84f9c7332c1a6af9a2a886c7ff7298be3a92e01d48ebfec296eba1" Sep 07 11:34:47.300990 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.300964 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" event={"ID":"2f88cef7-6702-4aa7-a455-afa2bc59e1c8","Type":"ContainerDied","Data":"29833c0625223fdba9afff16704efeb459903c5db3237aa08a4936baff7f32d2"} Sep 07 11:34:47.301094 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.301007 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7" Sep 07 11:34:47.306957 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.306939 2571 scope.go:117] "RemoveContainer" containerID="13b8301dcf98a9b7e35273d34dab32f55c54f9defc70927805f7bf5d45e7a75f" Sep 07 11:34:47.314626 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.314604 2571 scope.go:117] "RemoveContainer" containerID="44bd7999a25a6536cfbd367e57c0d9d82268f6722826e14194f48917ce97be94" Sep 07 11:34:47.321463 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.321400 2571 scope.go:117] "RemoveContainer" containerID="fb1faed985d05f42e43d23167fee1f48b6eaa8c05fa3d37459fbc9bec9ca2ba1" Sep 07 11:34:47.322335 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.322315 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw"] Sep 07 11:34:47.327126 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.327104 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4f5gw"] Sep 07 11:34:47.329266 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.329249 2571 scope.go:117] "RemoveContainer" containerID="295632a47579ace8e681e0ea56275b7519fdd315afd241f74803d7d0d108c788" Sep 07 11:34:47.335825 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.335810 2571 scope.go:117] "RemoveContainer" containerID="241e3b76c5f51cfde54e528126765832ca6e5cdb6c7d5cf634e4ab0d726dd02a" Sep 07 11:34:47.337942 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.337927 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7"] Sep 07 11:34:47.342394 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.342380 2571 scope.go:117] "RemoveContainer" containerID="a510a7911d069da22060db0d96bd234a1915f2f356ed33f1421b77b6b0c14d72" Sep 07 11:34:47.344599 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.344580 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-brms7"] Sep 07 11:34:47.349362 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.349347 2571 scope.go:117] "RemoveContainer" containerID="06e595780d46c5f37f588bd33daa2c59954e4b991c0b33ac2c5c5a588dbe5ac6" Sep 07 11:34:47.355806 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.355792 2571 scope.go:117] "RemoveContainer" containerID="3efcc2c8eb151c2493eed734bd06b16c5bee3462ba81ea48dfbc8a249f997123" Sep 07 11:34:47.362088 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.362072 2571 scope.go:117] "RemoveContainer" containerID="76737fdc63280ad1db8e0bf2352e75a437b7299a2e4089f7e023356ac34b45fa" Sep 07 11:34:47.368392 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:47.368374 2571 scope.go:117] "RemoveContainer" containerID="f4620386f133a92cfb9a04f082f74ddbcd151397f123b83ed59185617e5b863f" Sep 07 11:34:48.123290 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:48.123258 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" path="/var/lib/kubelet/pods/2f88cef7-6702-4aa7-a455-afa2bc59e1c8/volumes" Sep 07 11:34:48.123852 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:34:48.123837 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" path="/var/lib/kubelet/pods/533a46d7-e4eb-4b1b-bbed-4271bf401e68/volumes" Sep 07 11:35:28.110748 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:28.110718 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:35:28.113258 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:28.112957 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:35:38.038628 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038589 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz"] Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038883 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="feast-init" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038898 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="feast-init" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038907 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="registry" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038913 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="registry" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038923 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="feast-init" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038931 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="feast-init" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038939 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="online" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038944 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="online" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038950 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="ui" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038956 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="ui" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038971 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="feast-apply" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038977 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="feast-apply" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038984 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="feast-apply" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038989 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="feast-apply" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.038998 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="offline" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039003 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="offline" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039010 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="online" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039015 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="online" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039020 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="ui" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039025 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="ui" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039032 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="offline" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039039 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="offline" Sep 07 11:35:38.039076 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039083 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="ui" Sep 07 11:35:38.039797 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039092 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="ui" Sep 07 11:35:38.039797 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039097 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="offline" Sep 07 11:35:38.039797 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039103 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f88cef7-6702-4aa7-a455-afa2bc59e1c8" containerName="online" Sep 07 11:35:38.039797 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039109 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="online" Sep 07 11:35:38.039797 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039116 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="offline" Sep 07 11:35:38.039797 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.039125 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="533a46d7-e4eb-4b1b-bbed-4271bf401e68" containerName="registry" Sep 07 11:35:38.042343 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.042327 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.045166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.045138 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 07 11:35:38.045314 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.045187 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 07 11:35:38.045314 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.045217 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 07 11:35:38.045314 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.045265 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 07 11:35:38.045473 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.045461 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 07 11:35:38.045526 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.045495 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 07 11:35:38.045599 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.045533 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-z8gft\"" Sep 07 11:35:38.054166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.054138 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz"] Sep 07 11:35:38.129978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.129950 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.130138 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.129982 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.130138 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.130004 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcj5p\" (UniqueName: \"kubernetes.io/projected/f56104aa-c45e-4bec-a002-27a373fc7b21-kube-api-access-qcj5p\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.130138 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.130106 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.130245 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.130136 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f56104aa-c45e-4bec-a002-27a373fc7b21-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.130245 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.130189 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.231570 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.231503 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.231570 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.231573 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.231838 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:35:38.231662 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 07 11:35:38.231838 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:35:38.231738 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-ui-tls podName:f56104aa-c45e-4bec-a002-27a373fc7b21 nodeName:}" failed. No retries permitted until 2026-09-07 11:35:38.731723177 +0000 UTC m=+2111.206079534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-d6kwz" (UID: "f56104aa-c45e-4bec-a002-27a373fc7b21") : secret "feast-simple-feast-setup-ui-tls" not found Sep 07 11:35:38.231838 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.231673 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcj5p\" (UniqueName: \"kubernetes.io/projected/f56104aa-c45e-4bec-a002-27a373fc7b21-kube-api-access-qcj5p\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.231838 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.231809 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.231838 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.231838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f56104aa-c45e-4bec-a002-27a373fc7b21-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.232072 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.231864 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.232072 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:35:38.231900 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:35:38.232072 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:35:38.231946 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 07 11:35:38.232072 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:35:38.231960 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls podName:f56104aa-c45e-4bec-a002-27a373fc7b21 nodeName:}" failed. No retries permitted until 2026-09-07 11:35:38.731944857 +0000 UTC m=+2111.206301216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-d6kwz" (UID: "f56104aa-c45e-4bec-a002-27a373fc7b21") : secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:35:38.232072 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:35:38.231979 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-offline-tls podName:f56104aa-c45e-4bec-a002-27a373fc7b21 nodeName:}" failed. No retries permitted until 2026-09-07 11:35:38.731970772 +0000 UTC m=+2111.206327128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-d6kwz" (UID: "f56104aa-c45e-4bec-a002-27a373fc7b21") : secret "feast-simple-feast-setup-offline-tls" not found Sep 07 11:35:38.232228 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.232157 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f56104aa-c45e-4bec-a002-27a373fc7b21-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.234029 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.234007 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.240125 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.240103 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcj5p\" (UniqueName: \"kubernetes.io/projected/f56104aa-c45e-4bec-a002-27a373fc7b21-kube-api-access-qcj5p\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.736632 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.736592 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.736857 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.736675 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.736857 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.736713 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.736857 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:35:38.736818 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:35:38.737056 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:35:38.736892 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls podName:f56104aa-c45e-4bec-a002-27a373fc7b21 nodeName:}" failed. No retries permitted until 2026-09-07 11:35:39.736878762 +0000 UTC m=+2112.211235118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-d6kwz" (UID: "f56104aa-c45e-4bec-a002-27a373fc7b21") : secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:35:38.739173 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.739140 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:38.739319 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:38.739140 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:39.745320 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:39.745223 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:39.747605 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:39.747584 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-d6kwz\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:39.852126 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:39.852083 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:40.129842 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:40.126096 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz"] Sep 07 11:35:40.131517 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:35:40.131482 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf56104aa_c45e_4bec_a002_27a373fc7b21.slice/crio-ae42821ceef4137e435a52acd38623612319e239f918541a30a96d2c26f9d51a WatchSource:0}: Error finding container ae42821ceef4137e435a52acd38623612319e239f918541a30a96d2c26f9d51a: Status 404 returned error can't find the container with id ae42821ceef4137e435a52acd38623612319e239f918541a30a96d2c26f9d51a Sep 07 11:35:40.497124 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:40.497039 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerStarted","Data":"cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6"} Sep 07 11:35:40.497124 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:40.497089 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerStarted","Data":"ae42821ceef4137e435a52acd38623612319e239f918541a30a96d2c26f9d51a"} Sep 07 11:35:45.517495 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:45.517463 2571 generic.go:356] "Generic (PLEG): container finished" podID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerID="cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6" exitCode=0 Sep 07 11:35:45.517890 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:45.517535 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerDied","Data":"cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6"} Sep 07 11:35:46.524405 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:46.524370 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerStarted","Data":"c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a"} Sep 07 11:35:51.546897 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:51.546865 2571 generic.go:356] "Generic (PLEG): container finished" podID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerID="c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a" exitCode=0 Sep 07 11:35:51.547267 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:51.546937 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerDied","Data":"c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a"} Sep 07 11:35:52.557490 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:52.557444 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerStarted","Data":"449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b"} Sep 07 11:35:52.557987 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:52.557520 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerStarted","Data":"5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da"} Sep 07 11:35:52.557987 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:52.557543 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerStarted","Data":"1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce"} Sep 07 11:35:52.557987 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:52.557582 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerStarted","Data":"ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8"} Sep 07 11:35:52.585567 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:52.585481 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podStartSLOduration=14.585459756 podStartE2EDuration="14.585459756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:35:52.584414988 +0000 UTC m=+2125.058771367" watchObservedRunningTime="2026-09-07 11:35:52.585459756 +0000 UTC m=+2125.059816135" Sep 07 11:35:54.852882 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:54.852849 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:54.853421 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:54.853404 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:54.853532 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:54.853522 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:54.853870 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:54.853856 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:54.854961 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:54.854929 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" Sep 07 11:35:54.855178 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:54.854958 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.35:6571: connect: connection refused" Sep 07 11:35:54.855395 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:54.855002 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.35:8816: connect: connection refused" Sep 07 11:35:54.855505 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:54.855036 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Sep 07 11:35:57.853364 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:57.853323 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" Sep 07 11:35:57.853976 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:57.853321 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.35:8816: connect: connection refused" Sep 07 11:35:57.854209 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:57.853323 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.35:6571: connect: connection refused" Sep 07 11:35:57.854333 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:57.853434 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Sep 07 11:35:59.852465 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:59.852430 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:59.852830 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:59.852480 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:59.852830 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:59.852493 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:35:59.852830 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:35:59.852505 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:36:00.854326 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:36:00.854294 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:36:00.854827 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:36:00.854394 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:36:00.854827 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:36:00.854629 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:36:00.855245 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:36:00.855220 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:36:00.855357 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:36:00.855291 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:36:00.855589 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:36:00.855571 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:36:00.859594 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:36:00.859578 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:36:01.607634 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:36:01.607601 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:37:03.102538 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.102507 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz"] Sep 07 11:37:03.103079 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.102802 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="offline" containerID="cri-o://5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da" gracePeriod=30 Sep 07 11:37:03.103079 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.102804 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="ui" containerID="cri-o://449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b" gracePeriod=30 Sep 07 11:37:03.103079 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.102836 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="online" containerID="cri-o://1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce" gracePeriod=30 Sep 07 11:37:03.103079 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.102840 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="registry" containerID="cri-o://ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8" gracePeriod=30 Sep 07 11:37:03.535168 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.535133 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9"] Sep 07 11:37:03.538914 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.538893 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.542175 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.542151 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-clq58\"" Sep 07 11:37:03.555930 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.555897 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9"] Sep 07 11:37:03.607155 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.607102 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.607368 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.607257 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.607368 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.607316 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.607368 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.607342 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbd28\" (UniqueName: \"kubernetes.io/projected/bcfca9e7-9763-4147-93fd-900477ed37c7-kube-api-access-vbd28\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.607517 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.607377 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.607517 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.607447 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfca9e7-9763-4147-93fd-900477ed37c7-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.708407 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.708359 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.708621 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.708418 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.708621 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.708448 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd28\" (UniqueName: \"kubernetes.io/projected/bcfca9e7-9763-4147-93fd-900477ed37c7-kube-api-access-vbd28\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.708621 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.708477 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.708621 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.708502 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfca9e7-9763-4147-93fd-900477ed37c7-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.708621 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:37:03.708532 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 07 11:37:03.708621 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:37:03.708619 2571 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:37:03.708958 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:37:03.708633 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-online-tls podName:bcfca9e7-9763-4147-93fd-900477ed37c7 nodeName:}" failed. No retries permitted until 2026-09-07 11:37:04.20861441 +0000 UTC m=+2196.682970784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-online-tls") pod "feast-simple-feast-setup-57576b8fcf-fdmn9" (UID: "bcfca9e7-9763-4147-93fd-900477ed37c7") : secret "feast-simple-feast-setup-online-tls" not found Sep 07 11:37:03.708958 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:37:03.708668 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-registry-tls podName:bcfca9e7-9763-4147-93fd-900477ed37c7 nodeName:}" failed. No retries permitted until 2026-09-07 11:37:04.208652358 +0000 UTC m=+2196.683008734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-fdmn9" (UID: "bcfca9e7-9763-4147-93fd-900477ed37c7") : secret "feast-simple-feast-setup-registry-tls" not found Sep 07 11:37:03.708958 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.708537 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.709169 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.709007 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfca9e7-9763-4147-93fd-900477ed37c7-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.711500 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.711475 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.711666 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.711643 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.717807 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.717780 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbd28\" (UniqueName: \"kubernetes.io/projected/bcfca9e7-9763-4147-93fd-900477ed37c7-kube-api-access-vbd28\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:03.838172 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.838139 2571 generic.go:356] "Generic (PLEG): container finished" podID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerID="449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b" exitCode=0 Sep 07 11:37:03.838327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:03.838205 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerDied","Data":"449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b"} Sep 07 11:37:04.213265 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:04.213188 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:04.213265 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:04.213246 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:04.215562 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:04.215527 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:04.215674 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:04.215580 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fdmn9\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:04.449254 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:04.449214 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:04.720614 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:04.720587 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9"] Sep 07 11:37:04.722000 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:37:04.721970 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcfca9e7_9763_4147_93fd_900477ed37c7.slice/crio-78e965fa85e8595c9eb6e49405006cb203a8399d02ffff3850f8aaa2ae81f7df WatchSource:0}: Error finding container 78e965fa85e8595c9eb6e49405006cb203a8399d02ffff3850f8aaa2ae81f7df: Status 404 returned error can't find the container with id 78e965fa85e8595c9eb6e49405006cb203a8399d02ffff3850f8aaa2ae81f7df Sep 07 11:37:04.845664 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:04.845622 2571 generic.go:356] "Generic (PLEG): container finished" podID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerID="1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce" exitCode=0 Sep 07 11:37:04.845839 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:04.845689 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerDied","Data":"1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce"} Sep 07 11:37:04.847214 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:04.847192 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerStarted","Data":"dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e"} Sep 07 11:37:04.847322 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:04.847227 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerStarted","Data":"78e965fa85e8595c9eb6e49405006cb203a8399d02ffff3850f8aaa2ae81f7df"} Sep 07 11:37:09.867067 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:09.867036 2571 generic.go:356] "Generic (PLEG): container finished" podID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerID="dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e" exitCode=0 Sep 07 11:37:09.867430 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:09.867110 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerDied","Data":"dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e"} Sep 07 11:37:10.855314 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:10.855263 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" Sep 07 11:37:10.873002 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:10.872972 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerStarted","Data":"7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5"} Sep 07 11:37:11.605079 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:11.605031 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Sep 07 11:37:15.895470 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:15.895434 2571 generic.go:356] "Generic (PLEG): container finished" podID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerID="7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5" exitCode=0 Sep 07 11:37:15.895842 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:15.895485 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerDied","Data":"7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5"} Sep 07 11:37:16.905520 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:16.905476 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerStarted","Data":"873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0"} Sep 07 11:37:16.906023 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:16.905559 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerStarted","Data":"eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06"} Sep 07 11:37:16.906023 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:16.905584 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerStarted","Data":"050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925"} Sep 07 11:37:16.906023 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:16.905602 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerStarted","Data":"23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd"} Sep 07 11:37:16.932815 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:16.932754 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podStartSLOduration=13.932733147 podStartE2EDuration="13.932733147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:37:16.930312766 +0000 UTC m=+2209.404669145" watchObservedRunningTime="2026-09-07 11:37:16.932733147 +0000 UTC m=+2209.407089526" Sep 07 11:37:19.449576 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:19.449419 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:19.449576 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:19.449480 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:19.449576 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:19.449499 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:19.449576 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:19.449515 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:19.451942 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:19.451652 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Sep 07 11:37:19.451942 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:19.451667 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" Sep 07 11:37:19.451942 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:19.451767 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.36:8816: connect: connection refused" Sep 07 11:37:19.451942 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:19.451831 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 07 11:37:20.855422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:20.855369 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" Sep 07 11:37:21.605429 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:21.605379 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Sep 07 11:37:22.450158 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:22.450109 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.36:8816: connect: connection refused" Sep 07 11:37:22.450663 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:22.450108 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" Sep 07 11:37:22.450663 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:22.450206 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 07 11:37:22.450663 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:22.450468 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Sep 07 11:37:24.450238 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:24.450200 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:24.450634 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:24.450254 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:24.450634 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:24.450270 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:24.450634 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:24.450282 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:25.450916 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:25.450678 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:25.450916 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:25.450859 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:25.451451 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:25.450956 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:25.451636 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:25.451610 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:25.451758 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:25.451679 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:25.451758 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:25.451712 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:25.456090 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:25.456069 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:25.956482 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:25.956449 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:37:30.855562 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:30.855501 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" Sep 07 11:37:30.856080 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:30.855778 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:37:31.605184 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:31.605147 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" Sep 07 11:37:31.605422 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:31.605252 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:37:33.746667 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.746640 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:37:33.868623 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.868501 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-ui-tls\") pod \"f56104aa-c45e-4bec-a002-27a373fc7b21\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " Sep 07 11:37:33.868623 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.868544 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls\") pod \"f56104aa-c45e-4bec-a002-27a373fc7b21\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " Sep 07 11:37:33.868853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.868692 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcj5p\" (UniqueName: \"kubernetes.io/projected/f56104aa-c45e-4bec-a002-27a373fc7b21-kube-api-access-qcj5p\") pod \"f56104aa-c45e-4bec-a002-27a373fc7b21\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " Sep 07 11:37:33.868853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.868727 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f56104aa-c45e-4bec-a002-27a373fc7b21-feast-data\") pod \"f56104aa-c45e-4bec-a002-27a373fc7b21\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " Sep 07 11:37:33.868853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.868763 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-offline-tls\") pod \"f56104aa-c45e-4bec-a002-27a373fc7b21\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " Sep 07 11:37:33.868853 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.868836 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-online-tls\") pod \"f56104aa-c45e-4bec-a002-27a373fc7b21\" (UID: \"f56104aa-c45e-4bec-a002-27a373fc7b21\") " Sep 07 11:37:33.869287 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.869252 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56104aa-c45e-4bec-a002-27a373fc7b21-feast-data" (OuterVolumeSpecName: "feast-data") pod "f56104aa-c45e-4bec-a002-27a373fc7b21" (UID: "f56104aa-c45e-4bec-a002-27a373fc7b21"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:37:33.871030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.870993 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-online-tls" (OuterVolumeSpecName: "online-tls") pod "f56104aa-c45e-4bec-a002-27a373fc7b21" (UID: "f56104aa-c45e-4bec-a002-27a373fc7b21"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:37:33.871147 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.871117 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f56104aa-c45e-4bec-a002-27a373fc7b21" (UID: "f56104aa-c45e-4bec-a002-27a373fc7b21"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:37:33.871208 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.871156 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f56104aa-c45e-4bec-a002-27a373fc7b21" (UID: "f56104aa-c45e-4bec-a002-27a373fc7b21"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:37:33.871248 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.871204 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56104aa-c45e-4bec-a002-27a373fc7b21-kube-api-access-qcj5p" (OuterVolumeSpecName: "kube-api-access-qcj5p") pod "f56104aa-c45e-4bec-a002-27a373fc7b21" (UID: "f56104aa-c45e-4bec-a002-27a373fc7b21"). InnerVolumeSpecName "kube-api-access-qcj5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:37:33.871248 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.871202 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f56104aa-c45e-4bec-a002-27a373fc7b21" (UID: "f56104aa-c45e-4bec-a002-27a373fc7b21"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:37:33.969462 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.969421 2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-ui-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:37:33.969462 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.969454 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:37:33.969462 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.969466 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcj5p\" (UniqueName: \"kubernetes.io/projected/f56104aa-c45e-4bec-a002-27a373fc7b21-kube-api-access-qcj5p\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:37:33.969462 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.969475 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f56104aa-c45e-4bec-a002-27a373fc7b21-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:37:33.969767 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.969483 2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-offline-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:37:33.969767 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.969491 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f56104aa-c45e-4bec-a002-27a373fc7b21-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:37:33.987771 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.987734 2571 generic.go:356] "Generic (PLEG): container finished" podID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerID="5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da" exitCode=137 Sep 07 11:37:33.987771 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.987768 2571 generic.go:356] "Generic (PLEG): container finished" podID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerID="ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8" exitCode=137 Sep 07 11:37:33.987940 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.987813 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" Sep 07 11:37:33.987940 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.987821 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerDied","Data":"5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da"} Sep 07 11:37:33.987940 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.987870 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerDied","Data":"ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8"} Sep 07 11:37:33.987940 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.987884 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz" event={"ID":"f56104aa-c45e-4bec-a002-27a373fc7b21","Type":"ContainerDied","Data":"ae42821ceef4137e435a52acd38623612319e239f918541a30a96d2c26f9d51a"} Sep 07 11:37:33.987940 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.987901 2571 scope.go:117] "RemoveContainer" containerID="449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b" Sep 07 11:37:33.996476 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:33.996460 2571 scope.go:117] "RemoveContainer" containerID="5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da" Sep 07 11:37:34.003930 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.003906 2571 scope.go:117] "RemoveContainer" containerID="1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce" Sep 07 11:37:34.011436 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.011415 2571 scope.go:117] "RemoveContainer" containerID="ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8" Sep 07 11:37:34.013210 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.013188 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz"] Sep 07 11:37:34.017177 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.017151 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-d6kwz"] Sep 07 11:37:34.020678 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.020657 2571 scope.go:117] "RemoveContainer" containerID="c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a" Sep 07 11:37:34.028030 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.028013 2571 scope.go:117] "RemoveContainer" containerID="cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6" Sep 07 11:37:34.036734 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.036716 2571 scope.go:117] "RemoveContainer" containerID="449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b" Sep 07 11:37:34.037017 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:37:34.036996 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b\": container with ID starting with 449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b not found: ID does not exist" containerID="449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b" Sep 07 11:37:34.037081 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.037027 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b"} err="failed to get container status \"449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b\": rpc error: code = NotFound desc = could not find container \"449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b\": container with ID starting with 449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b not found: ID does not exist" Sep 07 11:37:34.037081 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.037047 2571 scope.go:117] "RemoveContainer" containerID="5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da" Sep 07 11:37:34.037320 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:37:34.037287 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da\": container with ID starting with 5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da not found: ID does not exist" containerID="5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da" Sep 07 11:37:34.037356 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.037327 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da"} err="failed to get container status \"5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da\": rpc error: code = NotFound desc = could not find container \"5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da\": container with ID starting with 5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da not found: ID does not exist" Sep 07 11:37:34.037356 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.037343 2571 scope.go:117] "RemoveContainer" containerID="1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce" Sep 07 11:37:34.037580 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:37:34.037558 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce\": container with ID starting with 1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce not found: ID does not exist" containerID="1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce" Sep 07 11:37:34.037685 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.037581 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce"} err="failed to get container status \"1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce\": rpc error: code = NotFound desc = could not find container \"1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce\": container with ID starting with 1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce not found: ID does not exist" Sep 07 11:37:34.037685 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.037594 2571 scope.go:117] "RemoveContainer" containerID="ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8" Sep 07 11:37:34.037859 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:37:34.037841 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8\": container with ID starting with ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8 not found: ID does not exist" containerID="ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8" Sep 07 11:37:34.037920 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.037876 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8"} err="failed to get container status \"ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8\": rpc error: code = NotFound desc = could not find container \"ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8\": container with ID starting with ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8 not found: ID does not exist" Sep 07 11:37:34.037920 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.037914 2571 scope.go:117] "RemoveContainer" containerID="c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a" Sep 07 11:37:34.038182 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:37:34.038165 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a\": container with ID starting with c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a not found: ID does not exist" containerID="c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a" Sep 07 11:37:34.038228 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.038185 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a"} err="failed to get container status \"c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a\": rpc error: code = NotFound desc = could not find container \"c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a\": container with ID starting with c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a not found: ID does not exist" Sep 07 11:37:34.038228 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.038200 2571 scope.go:117] "RemoveContainer" containerID="cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6" Sep 07 11:37:34.038465 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:37:34.038446 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6\": container with ID starting with cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6 not found: ID does not exist" containerID="cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6" Sep 07 11:37:34.038515 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.038471 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6"} err="failed to get container status \"cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6\": rpc error: code = NotFound desc = could not find container \"cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6\": container with ID starting with cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6 not found: ID does not exist" Sep 07 11:37:34.038515 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.038484 2571 scope.go:117] "RemoveContainer" containerID="449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b" Sep 07 11:37:34.038725 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.038710 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b"} err="failed to get container status \"449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b\": rpc error: code = NotFound desc = could not find container \"449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b\": container with ID starting with 449eb43d79e7450a15963c59c04bcdfd9095a4fe5657f9de26a7dfe5c1e5744b not found: ID does not exist" Sep 07 11:37:34.038771 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.038725 2571 scope.go:117] "RemoveContainer" containerID="5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da" Sep 07 11:37:34.038928 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.038907 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da"} err="failed to get container status \"5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da\": rpc error: code = NotFound desc = could not find container \"5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da\": container with ID starting with 5074c379ee451b3747633b67ed0e2d3465fd65ee26504b9a16ce8da19c8df2da not found: ID does not exist" Sep 07 11:37:34.038966 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.038929 2571 scope.go:117] "RemoveContainer" containerID="1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce" Sep 07 11:37:34.039136 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.039122 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce"} err="failed to get container status \"1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce\": rpc error: code = NotFound desc = could not find container \"1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce\": container with ID starting with 1869aca698fbf9e72b7d1ffa80b8211b48ba2ce1c872c7956ddf31cfec169cce not found: ID does not exist" Sep 07 11:37:34.039176 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.039138 2571 scope.go:117] "RemoveContainer" containerID="ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8" Sep 07 11:37:34.039325 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.039309 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8"} err="failed to get container status \"ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8\": rpc error: code = NotFound desc = could not find container \"ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8\": container with ID starting with ce5d045763dddfcd35f9ab1e583f1f7e5e0e980e41d527bf6d4346f1552040b8 not found: ID does not exist" Sep 07 11:37:34.039371 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.039325 2571 scope.go:117] "RemoveContainer" containerID="c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a" Sep 07 11:37:34.039506 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.039491 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a"} err="failed to get container status \"c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a\": rpc error: code = NotFound desc = could not find container \"c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a\": container with ID starting with c44f06034d20f961315f484dc708b22eea3c2e88a9e3c48f6bf8d5fe6ab9966a not found: ID does not exist" Sep 07 11:37:34.039569 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.039507 2571 scope.go:117] "RemoveContainer" containerID="cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6" Sep 07 11:37:34.039702 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.039687 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6"} err="failed to get container status \"cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6\": rpc error: code = NotFound desc = could not find container \"cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6\": container with ID starting with cd907590ada22954174f609b593b6c581218f65f1365c40e8d4fc121961c99c6 not found: ID does not exist" Sep 07 11:37:34.124473 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:37:34.124391 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" path="/var/lib/kubelet/pods/f56104aa-c45e-4bec-a002-27a373fc7b21/volumes" Sep 07 11:38:28.383343 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383308 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf"] Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383614 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="feast-init" Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383629 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="feast-init" Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383639 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="ui" Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383648 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="ui" Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383664 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="online" Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383672 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="online" Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383686 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="registry" Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383692 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="registry" Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383703 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="offline" Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383708 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="offline" Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383718 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="feast-apply" Sep 07 11:38:28.383740 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383722 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="feast-apply" Sep 07 11:38:28.384096 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383775 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="offline" Sep 07 11:38:28.384096 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383782 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="ui" Sep 07 11:38:28.384096 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383791 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="registry" Sep 07 11:38:28.384096 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.383796 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f56104aa-c45e-4bec-a002-27a373fc7b21" containerName="online" Sep 07 11:38:28.388135 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.388117 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.391008 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.390986 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 07 11:38:28.391183 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.391010 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 07 11:38:28.391183 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.391134 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 07 11:38:28.391321 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.391309 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 07 11:38:28.392146 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.392129 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 07 11:38:28.392210 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.392159 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 07 11:38:28.392210 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.392136 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-bzt9b\"" Sep 07 11:38:28.399592 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.399542 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf"] Sep 07 11:38:28.491332 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.491304 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.491484 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.491340 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.491484 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.491366 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/85a72771-f037-4cc1-96cc-bda8a70a6182-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.491484 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.491385 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/85a72771-f037-4cc1-96cc-bda8a70a6182-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.491484 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.491426 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.491484 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.491444 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpljs\" (UniqueName: \"kubernetes.io/projected/85a72771-f037-4cc1-96cc-bda8a70a6182-kube-api-access-jpljs\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.591781 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.591752 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.591946 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.591794 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.591946 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.591848 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/85a72771-f037-4cc1-96cc-bda8a70a6182-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.591946 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.591867 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/85a72771-f037-4cc1-96cc-bda8a70a6182-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.592059 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:38:28.592021 2571 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 07 11:38:28.592117 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.592068 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.592117 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:38:28.592095 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-ui-tls podName:85a72771-f037-4cc1-96cc-bda8a70a6182 nodeName:}" failed. No retries permitted until 2026-09-07 11:38:29.092073557 +0000 UTC m=+2281.566429914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" (UID: "85a72771-f037-4cc1-96cc-bda8a70a6182") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 07 11:38:28.592228 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.592134 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpljs\" (UniqueName: \"kubernetes.io/projected/85a72771-f037-4cc1-96cc-bda8a70a6182-kube-api-access-jpljs\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.592384 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.592364 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/85a72771-f037-4cc1-96cc-bda8a70a6182-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.592604 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.592588 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/85a72771-f037-4cc1-96cc-bda8a70a6182-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.594245 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.594228 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.594307 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.594271 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:28.604023 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:28.604002 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpljs\" (UniqueName: \"kubernetes.io/projected/85a72771-f037-4cc1-96cc-bda8a70a6182-kube-api-access-jpljs\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:29.096703 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:29.096665 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:29.098947 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:29.098924 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qqjgf\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:29.297123 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:29.297092 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:29.571227 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:29.571200 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf"] Sep 07 11:38:29.571981 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:38:29.571958 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85a72771_f037_4cc1_96cc_bda8a70a6182.slice/crio-54d207c0e70ab98fd0f0f91a487c9663041470cb92ceea23812f225e12f57e1c WatchSource:0}: Error finding container 54d207c0e70ab98fd0f0f91a487c9663041470cb92ceea23812f225e12f57e1c: Status 404 returned error can't find the container with id 54d207c0e70ab98fd0f0f91a487c9663041470cb92ceea23812f225e12f57e1c Sep 07 11:38:30.201773 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:30.201728 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerStarted","Data":"de6764c1a7060d376ad7e932e94ecf619b944551c66e67b0ebbe59b2e7c7d86d"} Sep 07 11:38:30.201991 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:30.201787 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerStarted","Data":"54d207c0e70ab98fd0f0f91a487c9663041470cb92ceea23812f225e12f57e1c"} Sep 07 11:38:35.222125 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:35.222093 2571 generic.go:356] "Generic (PLEG): container finished" podID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerID="de6764c1a7060d376ad7e932e94ecf619b944551c66e67b0ebbe59b2e7c7d86d" exitCode=0 Sep 07 11:38:35.222532 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:35.222163 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerDied","Data":"de6764c1a7060d376ad7e932e94ecf619b944551c66e67b0ebbe59b2e7c7d86d"} Sep 07 11:38:36.227930 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:36.227892 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerStarted","Data":"50f33d3298ad95fc27f3447f4361d3e2985acd0a479817ba5034f4d5f120ce60"} Sep 07 11:38:56.308194 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:56.308157 2571 generic.go:356] "Generic (PLEG): container finished" podID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerID="50f33d3298ad95fc27f3447f4361d3e2985acd0a479817ba5034f4d5f120ce60" exitCode=0 Sep 07 11:38:56.308582 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:56.308197 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerDied","Data":"50f33d3298ad95fc27f3447f4361d3e2985acd0a479817ba5034f4d5f120ce60"} Sep 07 11:38:57.316451 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:57.316413 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerStarted","Data":"5f3833f75c9a29faffbadccc61baee089d710655e30b792302e2a764bc111801"} Sep 07 11:38:57.316968 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:57.316466 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerStarted","Data":"0167c18c66bf9bfc1232f91e5995020000772462e94453145c16af5399d4b3e2"} Sep 07 11:38:57.316968 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:57.316485 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerStarted","Data":"084ca4c3658c6821a26b5c246354c71a50d11c3f7ee0ca14a1bb62d595fdf5f4"} Sep 07 11:38:57.348168 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:57.348103 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podStartSLOduration=29.348081343 podStartE2EDuration="29.348081343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:38:57.347823008 +0000 UTC m=+2309.822179385" watchObservedRunningTime="2026-09-07 11:38:57.348081343 +0000 UTC m=+2309.822437722" Sep 07 11:38:59.297984 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:59.297942 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:59.298473 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:59.298010 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:59.298473 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:59.298031 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:59.298473 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:59.298048 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:59.298473 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:59.298064 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:59.298738 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:59.298620 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:38:59.299495 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:59.299463 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 07 11:38:59.299639 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:59.299469 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Sep 07 11:38:59.299639 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:38:59.299474 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.37:8816: connect: connection refused" Sep 07 11:39:02.297951 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:02.297909 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.37:8816: connect: connection refused" Sep 07 11:39:02.298429 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:02.297908 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Sep 07 11:39:02.298429 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:02.298081 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 07 11:39:05.298284 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:05.298251 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:39:05.298884 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:05.298824 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:39:05.299041 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:05.299020 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:39:05.353260 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:05.353236 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:39:06.303688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:06.303648 2571 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 11:39:09.106632 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:09.106593 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:39:09.770974 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:09.770933 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:39:10.301704 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:10.301663 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 07 11:39:30.301811 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:30.301706 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 07 11:39:50.301699 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:39:50.301650 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 07 11:40:10.303672 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:10.303632 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 07 11:40:10.694048 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:10.693963 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf"] Sep 07 11:40:10.694229 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:10.694185 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" containerID="cri-o://084ca4c3658c6821a26b5c246354c71a50d11c3f7ee0ca14a1bb62d595fdf5f4" gracePeriod=30 Sep 07 11:40:10.694327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:10.694300 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="ui" containerID="cri-o://5f3833f75c9a29faffbadccc61baee089d710655e30b792302e2a764bc111801" gracePeriod=30 Sep 07 11:40:10.694410 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:10.694299 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="offline" containerID="cri-o://0167c18c66bf9bfc1232f91e5995020000772462e94453145c16af5399d4b3e2" gracePeriod=30 Sep 07 11:40:10.820010 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:10.819973 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9"] Sep 07 11:40:10.820256 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:10.820228 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="registry" containerID="cri-o://23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd" gracePeriod=30 Sep 07 11:40:10.820357 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:10.820279 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="ui" containerID="cri-o://873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0" gracePeriod=30 Sep 07 11:40:10.820357 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:10.820301 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="online" containerID="cri-o://050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925" gracePeriod=30 Sep 07 11:40:10.820477 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:10.820434 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="offline" containerID="cri-o://eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06" gracePeriod=30 Sep 07 11:40:11.611396 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:11.611357 2571 generic.go:356] "Generic (PLEG): container finished" podID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerID="873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0" exitCode=0 Sep 07 11:40:11.611801 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:11.611426 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerDied","Data":"873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0"} Sep 07 11:40:11.613941 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:11.613920 2571 generic.go:356] "Generic (PLEG): container finished" podID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerID="5f3833f75c9a29faffbadccc61baee089d710655e30b792302e2a764bc111801" exitCode=0 Sep 07 11:40:11.613941 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:11.613937 2571 generic.go:356] "Generic (PLEG): container finished" podID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerID="084ca4c3658c6821a26b5c246354c71a50d11c3f7ee0ca14a1bb62d595fdf5f4" exitCode=0 Sep 07 11:40:11.614086 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:11.613953 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerDied","Data":"5f3833f75c9a29faffbadccc61baee089d710655e30b792302e2a764bc111801"} Sep 07 11:40:11.614086 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:11.613989 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerDied","Data":"084ca4c3658c6821a26b5c246354c71a50d11c3f7ee0ca14a1bb62d595fdf5f4"} Sep 07 11:40:12.620526 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:12.620490 2571 generic.go:356] "Generic (PLEG): container finished" podID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerID="050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925" exitCode=0 Sep 07 11:40:12.620898 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:12.620563 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerDied","Data":"050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925"} Sep 07 11:40:15.353371 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:15.353334 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Sep 07 11:40:15.452103 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:15.452062 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" Sep 07 11:40:15.953699 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:15.953663 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 07 11:40:19.107570 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:19.107515 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 07 11:40:25.353297 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:25.353256 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Sep 07 11:40:25.451495 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:25.451461 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" Sep 07 11:40:25.953562 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:25.953514 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 07 11:40:28.136410 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:28.136310 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:40:28.140515 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:28.140217 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:40:29.107659 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:29.107612 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 07 11:40:35.353711 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:35.353671 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" Sep 07 11:40:35.354079 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:35.353790 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:40:35.452349 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:35.452307 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" Sep 07 11:40:35.452527 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:35.452436 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:40:35.953805 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:35.953768 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Sep 07 11:40:35.953961 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:35.953874 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:40:39.107647 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:39.107600 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Sep 07 11:40:39.108149 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:39.107755 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:40:40.733802 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:40.733778 2571 generic.go:356] "Generic (PLEG): container finished" podID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerID="0167c18c66bf9bfc1232f91e5995020000772462e94453145c16af5399d4b3e2" exitCode=137 Sep 07 11:40:40.734074 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:40.733848 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerDied","Data":"0167c18c66bf9bfc1232f91e5995020000772462e94453145c16af5399d4b3e2"} Sep 07 11:40:41.342927 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.342904 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:40:41.403142 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.403111 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpljs\" (UniqueName: \"kubernetes.io/projected/85a72771-f037-4cc1-96cc-bda8a70a6182-kube-api-access-jpljs\") pod \"85a72771-f037-4cc1-96cc-bda8a70a6182\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " Sep 07 11:40:41.403284 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.403166 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/85a72771-f037-4cc1-96cc-bda8a70a6182-feast-data\") pod \"85a72771-f037-4cc1-96cc-bda8a70a6182\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " Sep 07 11:40:41.403284 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.403240 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/85a72771-f037-4cc1-96cc-bda8a70a6182-registry-tls\") pod \"85a72771-f037-4cc1-96cc-bda8a70a6182\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " Sep 07 11:40:41.403284 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.403281 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-online-tls\") pod \"85a72771-f037-4cc1-96cc-bda8a70a6182\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " Sep 07 11:40:41.403472 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.403333 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-ui-tls\") pod \"85a72771-f037-4cc1-96cc-bda8a70a6182\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " Sep 07 11:40:41.403472 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.403359 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-offline-tls\") pod \"85a72771-f037-4cc1-96cc-bda8a70a6182\" (UID: \"85a72771-f037-4cc1-96cc-bda8a70a6182\") " Sep 07 11:40:41.404047 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.404001 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a72771-f037-4cc1-96cc-bda8a70a6182-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "85a72771-f037-4cc1-96cc-bda8a70a6182" (UID: "85a72771-f037-4cc1-96cc-bda8a70a6182"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 11:40:41.404497 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.404451 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a72771-f037-4cc1-96cc-bda8a70a6182-feast-data" (OuterVolumeSpecName: "feast-data") pod "85a72771-f037-4cc1-96cc-bda8a70a6182" (UID: "85a72771-f037-4cc1-96cc-bda8a70a6182"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:40:41.406220 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.406187 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "85a72771-f037-4cc1-96cc-bda8a70a6182" (UID: "85a72771-f037-4cc1-96cc-bda8a70a6182"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:40:41.406336 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.406310 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a72771-f037-4cc1-96cc-bda8a70a6182-kube-api-access-jpljs" (OuterVolumeSpecName: "kube-api-access-jpljs") pod "85a72771-f037-4cc1-96cc-bda8a70a6182" (UID: "85a72771-f037-4cc1-96cc-bda8a70a6182"). InnerVolumeSpecName "kube-api-access-jpljs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:40:41.406390 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.406326 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-online-tls" (OuterVolumeSpecName: "online-tls") pod "85a72771-f037-4cc1-96cc-bda8a70a6182" (UID: "85a72771-f037-4cc1-96cc-bda8a70a6182"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:40:41.406432 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.406400 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "85a72771-f037-4cc1-96cc-bda8a70a6182" (UID: "85a72771-f037-4cc1-96cc-bda8a70a6182"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:40:41.452926 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.452906 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:40:41.504762 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.504707 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-ui-tls\") pod \"bcfca9e7-9763-4147-93fd-900477ed37c7\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " Sep 07 11:40:41.504762 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.504737 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfca9e7-9763-4147-93fd-900477ed37c7-feast-data\") pod \"bcfca9e7-9763-4147-93fd-900477ed37c7\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " Sep 07 11:40:41.504900 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.504785 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbd28\" (UniqueName: \"kubernetes.io/projected/bcfca9e7-9763-4147-93fd-900477ed37c7-kube-api-access-vbd28\") pod \"bcfca9e7-9763-4147-93fd-900477ed37c7\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " Sep 07 11:40:41.504900 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.504810 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-offline-tls\") pod \"bcfca9e7-9763-4147-93fd-900477ed37c7\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " Sep 07 11:40:41.504900 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.504838 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-online-tls\") pod \"bcfca9e7-9763-4147-93fd-900477ed37c7\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " Sep 07 11:40:41.504900 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.504867 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-registry-tls\") pod \"bcfca9e7-9763-4147-93fd-900477ed37c7\" (UID: \"bcfca9e7-9763-4147-93fd-900477ed37c7\") " Sep 07 11:40:41.505104 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.505076 2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-ui-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.505104 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.505094 2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-offline-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.505222 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.505108 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpljs\" (UniqueName: \"kubernetes.io/projected/85a72771-f037-4cc1-96cc-bda8a70a6182-kube-api-access-jpljs\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.505222 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.505121 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/85a72771-f037-4cc1-96cc-bda8a70a6182-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.505222 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.505137 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/85a72771-f037-4cc1-96cc-bda8a70a6182-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.505222 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.505153 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/85a72771-f037-4cc1-96cc-bda8a70a6182-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.505470 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.505346 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfca9e7-9763-4147-93fd-900477ed37c7-feast-data" (OuterVolumeSpecName: "feast-data") pod "bcfca9e7-9763-4147-93fd-900477ed37c7" (UID: "bcfca9e7-9763-4147-93fd-900477ed37c7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:40:41.506819 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.506794 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "bcfca9e7-9763-4147-93fd-900477ed37c7" (UID: "bcfca9e7-9763-4147-93fd-900477ed37c7"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:40:41.506894 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.506832 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "bcfca9e7-9763-4147-93fd-900477ed37c7" (UID: "bcfca9e7-9763-4147-93fd-900477ed37c7"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:40:41.506894 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.506845 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-online-tls" (OuterVolumeSpecName: "online-tls") pod "bcfca9e7-9763-4147-93fd-900477ed37c7" (UID: "bcfca9e7-9763-4147-93fd-900477ed37c7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:40:41.507200 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.507171 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bcfca9e7-9763-4147-93fd-900477ed37c7" (UID: "bcfca9e7-9763-4147-93fd-900477ed37c7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:40:41.507300 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.507181 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfca9e7-9763-4147-93fd-900477ed37c7-kube-api-access-vbd28" (OuterVolumeSpecName: "kube-api-access-vbd28") pod "bcfca9e7-9763-4147-93fd-900477ed37c7" (UID: "bcfca9e7-9763-4147-93fd-900477ed37c7"). InnerVolumeSpecName "kube-api-access-vbd28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:40:41.605753 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.605735 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbd28\" (UniqueName: \"kubernetes.io/projected/bcfca9e7-9763-4147-93fd-900477ed37c7-kube-api-access-vbd28\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.605753 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.605752 2571 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-offline-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.605868 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.605761 2571 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-online-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.605868 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.605768 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-registry-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.605868 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.605775 2571 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bcfca9e7-9763-4147-93fd-900477ed37c7-ui-tls\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.605868 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.605782 2571 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfca9e7-9763-4147-93fd-900477ed37c7-feast-data\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:40:41.740093 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.740068 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" event={"ID":"85a72771-f037-4cc1-96cc-bda8a70a6182","Type":"ContainerDied","Data":"54d207c0e70ab98fd0f0f91a487c9663041470cb92ceea23812f225e12f57e1c"} Sep 07 11:40:41.740454 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.740095 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf" Sep 07 11:40:41.740454 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.740123 2571 scope.go:117] "RemoveContainer" containerID="5f3833f75c9a29faffbadccc61baee089d710655e30b792302e2a764bc111801" Sep 07 11:40:41.743245 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.743227 2571 generic.go:356] "Generic (PLEG): container finished" podID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerID="eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06" exitCode=137 Sep 07 11:40:41.743245 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.743244 2571 generic.go:356] "Generic (PLEG): container finished" podID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerID="23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd" exitCode=137 Sep 07 11:40:41.743385 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.743297 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerDied","Data":"eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06"} Sep 07 11:40:41.743385 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.743321 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerDied","Data":"23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd"} Sep 07 11:40:41.743385 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.743328 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" Sep 07 11:40:41.743491 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.743334 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9" event={"ID":"bcfca9e7-9763-4147-93fd-900477ed37c7","Type":"ContainerDied","Data":"78e965fa85e8595c9eb6e49405006cb203a8399d02ffff3850f8aaa2ae81f7df"} Sep 07 11:40:41.748933 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.748917 2571 scope.go:117] "RemoveContainer" containerID="0167c18c66bf9bfc1232f91e5995020000772462e94453145c16af5399d4b3e2" Sep 07 11:40:41.757155 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.757138 2571 scope.go:117] "RemoveContainer" containerID="084ca4c3658c6821a26b5c246354c71a50d11c3f7ee0ca14a1bb62d595fdf5f4" Sep 07 11:40:41.764178 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.764161 2571 scope.go:117] "RemoveContainer" containerID="50f33d3298ad95fc27f3447f4361d3e2985acd0a479817ba5034f4d5f120ce60" Sep 07 11:40:41.767803 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.767785 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf"] Sep 07 11:40:41.771014 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.770984 2571 scope.go:117] "RemoveContainer" containerID="de6764c1a7060d376ad7e932e94ecf619b944551c66e67b0ebbe59b2e7c7d86d" Sep 07 11:40:41.773228 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.773211 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qqjgf"] Sep 07 11:40:41.778102 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.778084 2571 scope.go:117] "RemoveContainer" containerID="873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0" Sep 07 11:40:41.784727 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.784707 2571 scope.go:117] "RemoveContainer" containerID="eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06" Sep 07 11:40:41.784812 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.784798 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9"] Sep 07 11:40:41.790822 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.790801 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fdmn9"] Sep 07 11:40:41.791660 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.791644 2571 scope.go:117] "RemoveContainer" containerID="050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925" Sep 07 11:40:41.798072 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.798053 2571 scope.go:117] "RemoveContainer" containerID="23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd" Sep 07 11:40:41.804691 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.804671 2571 scope.go:117] "RemoveContainer" containerID="7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5" Sep 07 11:40:41.810910 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.810895 2571 scope.go:117] "RemoveContainer" containerID="dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e" Sep 07 11:40:41.817293 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.817278 2571 scope.go:117] "RemoveContainer" containerID="873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0" Sep 07 11:40:41.817509 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:40:41.817495 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0\": container with ID starting with 873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0 not found: ID does not exist" containerID="873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0" Sep 07 11:40:41.817577 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.817515 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0"} err="failed to get container status \"873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0\": rpc error: code = NotFound desc = could not find container \"873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0\": container with ID starting with 873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0 not found: ID does not exist" Sep 07 11:40:41.817577 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.817529 2571 scope.go:117] "RemoveContainer" containerID="eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06" Sep 07 11:40:41.817907 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:40:41.817889 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06\": container with ID starting with eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06 not found: ID does not exist" containerID="eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06" Sep 07 11:40:41.817958 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.817915 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06"} err="failed to get container status \"eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06\": rpc error: code = NotFound desc = could not find container \"eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06\": container with ID starting with eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06 not found: ID does not exist" Sep 07 11:40:41.817958 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.817932 2571 scope.go:117] "RemoveContainer" containerID="050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925" Sep 07 11:40:41.818155 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:40:41.818141 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925\": container with ID starting with 050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925 not found: ID does not exist" containerID="050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925" Sep 07 11:40:41.818195 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.818157 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925"} err="failed to get container status \"050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925\": rpc error: code = NotFound desc = could not find container \"050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925\": container with ID starting with 050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925 not found: ID does not exist" Sep 07 11:40:41.818195 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.818170 2571 scope.go:117] "RemoveContainer" containerID="23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd" Sep 07 11:40:41.818406 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:40:41.818391 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd\": container with ID starting with 23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd not found: ID does not exist" containerID="23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd" Sep 07 11:40:41.818453 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.818410 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd"} err="failed to get container status \"23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd\": rpc error: code = NotFound desc = could not find container \"23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd\": container with ID starting with 23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd not found: ID does not exist" Sep 07 11:40:41.818453 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.818423 2571 scope.go:117] "RemoveContainer" containerID="7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5" Sep 07 11:40:41.818643 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:40:41.818630 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5\": container with ID starting with 7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5 not found: ID does not exist" containerID="7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5" Sep 07 11:40:41.818690 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.818646 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5"} err="failed to get container status \"7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5\": rpc error: code = NotFound desc = could not find container \"7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5\": container with ID starting with 7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5 not found: ID does not exist" Sep 07 11:40:41.818690 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.818657 2571 scope.go:117] "RemoveContainer" containerID="dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e" Sep 07 11:40:41.818877 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:40:41.818862 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e\": container with ID starting with dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e not found: ID does not exist" containerID="dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e" Sep 07 11:40:41.818921 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.818880 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e"} err="failed to get container status \"dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e\": rpc error: code = NotFound desc = could not find container \"dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e\": container with ID starting with dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e not found: ID does not exist" Sep 07 11:40:41.818921 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.818892 2571 scope.go:117] "RemoveContainer" containerID="873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0" Sep 07 11:40:41.819117 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.819097 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0"} err="failed to get container status \"873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0\": rpc error: code = NotFound desc = could not find container \"873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0\": container with ID starting with 873868794c0b8e7986760e9b4094dd02963777c89213380c0ab481ddd98c9fb0 not found: ID does not exist" Sep 07 11:40:41.819184 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.819128 2571 scope.go:117] "RemoveContainer" containerID="eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06" Sep 07 11:40:41.819369 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.819354 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06"} err="failed to get container status \"eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06\": rpc error: code = NotFound desc = could not find container \"eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06\": container with ID starting with eb7d911886894cb435bca23877a2c35be5040a506574cb37a49cd7ec4c0f3b06 not found: ID does not exist" Sep 07 11:40:41.819417 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.819369 2571 scope.go:117] "RemoveContainer" containerID="050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925" Sep 07 11:40:41.819595 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.819569 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925"} err="failed to get container status \"050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925\": rpc error: code = NotFound desc = could not find container \"050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925\": container with ID starting with 050c2a6af59f50e16321a64be3831f7fed36579cea239004a9f7b7c9f5c8c925 not found: ID does not exist" Sep 07 11:40:41.819686 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.819595 2571 scope.go:117] "RemoveContainer" containerID="23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd" Sep 07 11:40:41.819849 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.819832 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd"} err="failed to get container status \"23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd\": rpc error: code = NotFound desc = could not find container \"23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd\": container with ID starting with 23811d8abdd33f70c8473234af6edfdc96eacc88abb01d387e52c9f17cd8afdd not found: ID does not exist" Sep 07 11:40:41.819899 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.819850 2571 scope.go:117] "RemoveContainer" containerID="7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5" Sep 07 11:40:41.820069 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.820054 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5"} err="failed to get container status \"7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5\": rpc error: code = NotFound desc = could not find container \"7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5\": container with ID starting with 7a1fdcb2b651bace47f2aecc0184275f2869dedd1ab2f87e01bab8f4a2b161b5 not found: ID does not exist" Sep 07 11:40:41.820069 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.820069 2571 scope.go:117] "RemoveContainer" containerID="dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e" Sep 07 11:40:41.820251 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:41.820235 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e"} err="failed to get container status \"dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e\": rpc error: code = NotFound desc = could not find container \"dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e\": container with ID starting with dd4c56e03b31760127d033fb141de89dff399299ff53bf919b1025a1c50aa12e not found: ID does not exist" Sep 07 11:40:42.123269 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:42.123225 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" path="/var/lib/kubelet/pods/85a72771-f037-4cc1-96cc-bda8a70a6182/volumes" Sep 07 11:40:42.123839 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:42.123817 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" path="/var/lib/kubelet/pods/bcfca9e7-9763-4147-93fd-900477ed37c7/volumes" Sep 07 11:40:59.907647 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.907606 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k7rbc/must-gather-5g7wt"] Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.907996 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="ui" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908013 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="ui" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908032 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="registry" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908041 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="registry" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908055 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="online" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908063 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="online" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908074 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="feast-init" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908083 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="feast-init" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908092 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="offline" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908099 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="offline" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908108 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="feast-apply" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908116 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="feast-apply" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908126 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="feast-init" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908134 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="feast-init" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908143 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="offline" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908151 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="offline" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908158 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="ui" Sep 07 11:40:59.908166 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908166 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="ui" Sep 07 11:40:59.909046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908180 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="feast-apply" Sep 07 11:40:59.909046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908187 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="feast-apply" Sep 07 11:40:59.909046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908196 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" Sep 07 11:40:59.909046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908207 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" Sep 07 11:40:59.909046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908302 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="ui" Sep 07 11:40:59.909046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908315 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="offline" Sep 07 11:40:59.909046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908327 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="offline" Sep 07 11:40:59.909046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908336 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="ui" Sep 07 11:40:59.909046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908346 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="registry" Sep 07 11:40:59.909046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908354 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcfca9e7-9763-4147-93fd-900477ed37c7" containerName="online" Sep 07 11:40:59.909046 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.908363 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a72771-f037-4cc1-96cc-bda8a70a6182" containerName="online" Sep 07 11:40:59.913495 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.913474 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k7rbc/must-gather-5g7wt" Sep 07 11:40:59.916305 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.916283 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k7rbc\"/\"default-dockercfg-7shv8\"" Sep 07 11:40:59.916425 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.916283 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k7rbc\"/\"kube-root-ca.crt\"" Sep 07 11:40:59.917632 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.917617 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k7rbc\"/\"openshift-service-ca.crt\"" Sep 07 11:40:59.921864 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:40:59.921839 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k7rbc/must-gather-5g7wt"] Sep 07 11:41:00.030223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:00.030180 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-must-gather-output\") pod \"must-gather-5g7wt\" (UID: \"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222\") " pod="openshift-must-gather-k7rbc/must-gather-5g7wt" Sep 07 11:41:00.030398 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:00.030235 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzp7j\" (UniqueName: \"kubernetes.io/projected/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-kube-api-access-dzp7j\") pod \"must-gather-5g7wt\" (UID: \"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222\") " pod="openshift-must-gather-k7rbc/must-gather-5g7wt" Sep 07 11:41:00.131066 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:00.131029 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-must-gather-output\") pod \"must-gather-5g7wt\" (UID: \"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222\") " pod="openshift-must-gather-k7rbc/must-gather-5g7wt" Sep 07 11:41:00.131241 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:00.131088 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzp7j\" (UniqueName: \"kubernetes.io/projected/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-kube-api-access-dzp7j\") pod \"must-gather-5g7wt\" (UID: \"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222\") " pod="openshift-must-gather-k7rbc/must-gather-5g7wt" Sep 07 11:41:00.131394 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:00.131370 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-must-gather-output\") pod \"must-gather-5g7wt\" (UID: \"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222\") " pod="openshift-must-gather-k7rbc/must-gather-5g7wt" Sep 07 11:41:00.139901 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:00.139874 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzp7j\" (UniqueName: \"kubernetes.io/projected/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-kube-api-access-dzp7j\") pod \"must-gather-5g7wt\" (UID: \"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222\") " pod="openshift-must-gather-k7rbc/must-gather-5g7wt" Sep 07 11:41:00.223395 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:00.223328 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k7rbc/must-gather-5g7wt" Sep 07 11:41:00.492373 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:00.492138 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k7rbc/must-gather-5g7wt"] Sep 07 11:41:00.494970 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:41:00.494942 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ec6ac6e_d315_4e7d_99d3_0b7577ef3222.slice/crio-4d98c755047a3fb4dfdff2cabfe3d1deb89b075d05e49486f3d884a49fb58091 WatchSource:0}: Error finding container 4d98c755047a3fb4dfdff2cabfe3d1deb89b075d05e49486f3d884a49fb58091: Status 404 returned error can't find the container with id 4d98c755047a3fb4dfdff2cabfe3d1deb89b075d05e49486f3d884a49fb58091 Sep 07 11:41:00.496780 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:00.496762 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 11:41:00.820168 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:00.820131 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k7rbc/must-gather-5g7wt" event={"ID":"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222","Type":"ContainerStarted","Data":"4d98c755047a3fb4dfdff2cabfe3d1deb89b075d05e49486f3d884a49fb58091"} Sep 07 11:41:06.851308 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:06.851118 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k7rbc/must-gather-5g7wt" event={"ID":"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222","Type":"ContainerStarted","Data":"b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944"} Sep 07 11:41:06.851308 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:06.851178 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k7rbc/must-gather-5g7wt" event={"ID":"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222","Type":"ContainerStarted","Data":"c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0"} Sep 07 11:41:06.869071 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:06.868973 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k7rbc/must-gather-5g7wt" podStartSLOduration=2.01409633 podStartE2EDuration="7.868952819s" totalImagesPullingTime="5.854856489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:41:06.867735818 +0000 UTC m=+2439.342092197" watchObservedRunningTime="2026-09-07 11:41:06.868952819 +0000 UTC m=+2439.343309198" Sep 07 11:41:15.887308 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:15.887206 2571 generic.go:356] "Generic (PLEG): container finished" podID="7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" containerID="c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0" exitCode=0 Sep 07 11:41:15.887308 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:15.887285 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k7rbc/must-gather-5g7wt" event={"ID":"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222","Type":"ContainerDied","Data":"c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0"} Sep 07 11:41:15.887904 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:15.887688 2571 scope.go:117] "RemoveContainer" containerID="c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0" Sep 07 11:41:16.633994 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:16.633964 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k7rbc_must-gather-5g7wt_7ec6ac6e-d315-4e7d-99d3-0b7577ef3222/gather/0.log" Sep 07 11:41:17.116388 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.116355 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zzq24/must-gather-ggtjc"] Sep 07 11:41:17.119323 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.119305 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzq24/must-gather-ggtjc" Sep 07 11:41:17.122081 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.122061 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zzq24\"/\"default-dockercfg-vzrfk\"" Sep 07 11:41:17.122221 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.122170 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzq24\"/\"openshift-service-ca.crt\"" Sep 07 11:41:17.123289 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.123272 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzq24\"/\"kube-root-ca.crt\"" Sep 07 11:41:17.131033 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.131013 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzq24/must-gather-ggtjc"] Sep 07 11:41:17.278285 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.278252 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/606f312e-38d2-4c28-920c-c2665eb16d37-must-gather-output\") pod \"must-gather-ggtjc\" (UID: \"606f312e-38d2-4c28-920c-c2665eb16d37\") " pod="openshift-must-gather-zzq24/must-gather-ggtjc" Sep 07 11:41:17.278447 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.278295 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wncc2\" (UniqueName: \"kubernetes.io/projected/606f312e-38d2-4c28-920c-c2665eb16d37-kube-api-access-wncc2\") pod \"must-gather-ggtjc\" (UID: \"606f312e-38d2-4c28-920c-c2665eb16d37\") " pod="openshift-must-gather-zzq24/must-gather-ggtjc" Sep 07 11:41:17.379248 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.379168 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/606f312e-38d2-4c28-920c-c2665eb16d37-must-gather-output\") pod \"must-gather-ggtjc\" (UID: \"606f312e-38d2-4c28-920c-c2665eb16d37\") " pod="openshift-must-gather-zzq24/must-gather-ggtjc" Sep 07 11:41:17.379248 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.379210 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wncc2\" (UniqueName: \"kubernetes.io/projected/606f312e-38d2-4c28-920c-c2665eb16d37-kube-api-access-wncc2\") pod \"must-gather-ggtjc\" (UID: \"606f312e-38d2-4c28-920c-c2665eb16d37\") " pod="openshift-must-gather-zzq24/must-gather-ggtjc" Sep 07 11:41:17.379517 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.379498 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/606f312e-38d2-4c28-920c-c2665eb16d37-must-gather-output\") pod \"must-gather-ggtjc\" (UID: \"606f312e-38d2-4c28-920c-c2665eb16d37\") " pod="openshift-must-gather-zzq24/must-gather-ggtjc" Sep 07 11:41:17.388140 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.388116 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wncc2\" (UniqueName: \"kubernetes.io/projected/606f312e-38d2-4c28-920c-c2665eb16d37-kube-api-access-wncc2\") pod \"must-gather-ggtjc\" (UID: \"606f312e-38d2-4c28-920c-c2665eb16d37\") " pod="openshift-must-gather-zzq24/must-gather-ggtjc" Sep 07 11:41:17.427300 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.427267 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzq24/must-gather-ggtjc" Sep 07 11:41:17.695128 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.695042 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzq24/must-gather-ggtjc"] Sep 07 11:41:17.698632 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:41:17.698607 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod606f312e_38d2_4c28_920c_c2665eb16d37.slice/crio-a34f43a6813b844f2aab43b4fc551a57aae45304eb9a89d8cb5af8fe18575f73 WatchSource:0}: Error finding container a34f43a6813b844f2aab43b4fc551a57aae45304eb9a89d8cb5af8fe18575f73: Status 404 returned error can't find the container with id a34f43a6813b844f2aab43b4fc551a57aae45304eb9a89d8cb5af8fe18575f73 Sep 07 11:41:17.895790 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:17.895758 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzq24/must-gather-ggtjc" event={"ID":"606f312e-38d2-4c28-920c-c2665eb16d37","Type":"ContainerStarted","Data":"a34f43a6813b844f2aab43b4fc551a57aae45304eb9a89d8cb5af8fe18575f73"} Sep 07 11:41:18.902729 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:18.902679 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzq24/must-gather-ggtjc" event={"ID":"606f312e-38d2-4c28-920c-c2665eb16d37","Type":"ContainerStarted","Data":"d3954bf6a57e4c26b5c2c052e6e868224f04b22d951581c157fc96559e71a281"} Sep 07 11:41:18.903223 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:18.902747 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzq24/must-gather-ggtjc" event={"ID":"606f312e-38d2-4c28-920c-c2665eb16d37","Type":"ContainerStarted","Data":"d45748fe0304135058942b8ead537346a2351e019b4889e8e144d4e52efdc862"} Sep 07 11:41:18.920464 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:18.920236 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zzq24/must-gather-ggtjc" podStartSLOduration=1.1481032 podStartE2EDuration="1.920217168s" totalImagesPullingTime="772.113968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:41:18.918435369 +0000 UTC m=+2451.392791746" watchObservedRunningTime="2026-09-07 11:41:18.920217168 +0000 UTC m=+2451.394573548" Sep 07 11:41:19.756095 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:19.756047 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4vbhj_7fbcd4d6-16cc-4ba2-933f-6938932d59a3/global-pull-secret-syncer/0.log" Sep 07 11:41:19.949333 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:19.949297 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p2db7_36904d67-6a0b-4530-bfda-f87040d0dd71/konnectivity-agent/0.log" Sep 07 11:41:20.014387 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:20.014350 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-34.ec2.internal_77bb3ed25ee7e95473707df13e75eca1/haproxy/0.log" Sep 07 11:41:21.944974 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:21.944927 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-k7rbc/must-gather-5g7wt"] Sep 07 11:41:21.945450 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:21.945171 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-k7rbc/must-gather-5g7wt" podUID="7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" containerName="copy" containerID="cri-o://b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944" gracePeriod=2 Sep 07 11:41:21.985157 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:41:21.948051 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5g7wt\" is forbidden: User \"system:node:ip-10-0-138-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-k7rbc\": no relationship found between node 'ip-10-0-138-34.ec2.internal' and this object" podUID="7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" pod="openshift-must-gather-k7rbc/must-gather-5g7wt" Sep 07 11:41:21.985157 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:21.983687 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-k7rbc/must-gather-5g7wt"] Sep 07 11:41:22.396872 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.396820 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k7rbc_must-gather-5g7wt_7ec6ac6e-d315-4e7d-99d3-0b7577ef3222/copy/0.log" Sep 07 11:41:22.401868 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.401805 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k7rbc/must-gather-5g7wt" Sep 07 11:41:22.439649 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.439612 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-must-gather-output\") pod \"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222\" (UID: \"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222\") " Sep 07 11:41:22.439903 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.439884 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzp7j\" (UniqueName: \"kubernetes.io/projected/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-kube-api-access-dzp7j\") pod \"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222\" (UID: \"7ec6ac6e-d315-4e7d-99d3-0b7577ef3222\") " Sep 07 11:41:22.441487 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.440295 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" (UID: "7ec6ac6e-d315-4e7d-99d3-0b7577ef3222"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 11:41:22.441835 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.441807 2571 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-must-gather-output\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:41:22.445689 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.445657 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-kube-api-access-dzp7j" (OuterVolumeSpecName: "kube-api-access-dzp7j") pod "7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" (UID: "7ec6ac6e-d315-4e7d-99d3-0b7577ef3222"). InnerVolumeSpecName "kube-api-access-dzp7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:41:22.542467 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.542424 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzp7j\" (UniqueName: \"kubernetes.io/projected/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222-kube-api-access-dzp7j\") on node \"ip-10-0-138-34.ec2.internal\" DevicePath \"\"" Sep 07 11:41:22.926451 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.926360 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k7rbc_must-gather-5g7wt_7ec6ac6e-d315-4e7d-99d3-0b7577ef3222/copy/0.log" Sep 07 11:41:22.926883 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.926854 2571 generic.go:356] "Generic (PLEG): container finished" podID="7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" containerID="b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944" exitCode=143 Sep 07 11:41:22.927041 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.927015 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k7rbc/must-gather-5g7wt" Sep 07 11:41:22.927117 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.927040 2571 scope.go:117] "RemoveContainer" containerID="b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944" Sep 07 11:41:22.939062 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.938968 2571 scope.go:117] "RemoveContainer" containerID="c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0" Sep 07 11:41:22.955994 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.954385 2571 scope.go:117] "RemoveContainer" containerID="b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944" Sep 07 11:41:22.955994 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:41:22.954771 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944\": container with ID starting with b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944 not found: ID does not exist" containerID="b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944" Sep 07 11:41:22.955994 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.954807 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944"} err="failed to get container status \"b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944\": rpc error: code = NotFound desc = could not find container \"b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944\": container with ID starting with b773f269418d37031f464032c371221f821dafec80f7e2f6053b99ce7d2cc944 not found: ID does not exist" Sep 07 11:41:22.955994 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.954832 2571 scope.go:117] "RemoveContainer" containerID="c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0" Sep 07 11:41:22.955994 ip-10-0-138-34 kubenswrapper[2571]: E0907 11:41:22.955080 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0\": container with ID starting with c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0 not found: ID does not exist" containerID="c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0" Sep 07 11:41:22.955994 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:22.955105 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0"} err="failed to get container status \"c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0\": rpc error: code = NotFound desc = could not find container \"c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0\": container with ID starting with c721ae03abd51da50051fb8540745dd4708659f1e8911aa251edcfb07c3c36d0 not found: ID does not exist" Sep 07 11:41:23.817002 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:23.816965 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-svn6v_70eb15f4-2ce4-4ceb-9113-203baff5b63c/kube-state-metrics/0.log" Sep 07 11:41:23.848919 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:23.848888 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-svn6v_70eb15f4-2ce4-4ceb-9113-203baff5b63c/kube-rbac-proxy-main/0.log" Sep 07 11:41:23.879663 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:23.879635 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-svn6v_70eb15f4-2ce4-4ceb-9113-203baff5b63c/kube-rbac-proxy-self/0.log" Sep 07 11:41:23.910078 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:23.910043 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-cc58458f6-sd77f_39c89bda-7d1b-4a28-8f38-ce22fc364f16/metrics-server/0.log" Sep 07 11:41:23.949390 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:23.949310 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-m5tkd_9593324d-c084-43ab-a4cd-c47036faff90/monitoring-plugin/0.log" Sep 07 11:41:24.060327 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:24.060296 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p67qs_d734a792-d188-4c32-8919-122260339217/node-exporter/0.log" Sep 07 11:41:24.083389 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:24.083287 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p67qs_d734a792-d188-4c32-8919-122260339217/kube-rbac-proxy/0.log" Sep 07 11:41:24.108304 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:24.108274 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p67qs_d734a792-d188-4c32-8919-122260339217/init-textfile/0.log" Sep 07 11:41:24.126613 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:24.126581 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" path="/var/lib/kubelet/pods/7ec6ac6e-d315-4e7d-99d3-0b7577ef3222/volumes" Sep 07 11:41:24.208860 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:24.208827 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2mz5g_18ba05e2-441f-4905-a567-a93fccc511de/kube-rbac-proxy-main/0.log" Sep 07 11:41:24.233681 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:24.233628 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2mz5g_18ba05e2-441f-4905-a567-a93fccc511de/kube-rbac-proxy-self/0.log" Sep 07 11:41:24.256012 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:24.255985 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2mz5g_18ba05e2-441f-4905-a567-a93fccc511de/openshift-state-metrics/0.log" Sep 07 11:41:24.578441 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:24.578408 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-96b5d9885-tst5d_5d0742af-561c-4c78-8b4b-9198b3d980fb/telemeter-client/0.log" Sep 07 11:41:24.605682 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:24.605650 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-96b5d9885-tst5d_5d0742af-561c-4c78-8b4b-9198b3d980fb/reload/0.log" Sep 07 11:41:24.640108 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:24.640061 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-96b5d9885-tst5d_5d0742af-561c-4c78-8b4b-9198b3d980fb/kube-rbac-proxy/0.log" Sep 07 11:41:26.448982 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:26.448527 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/2.log" Sep 07 11:41:26.456589 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:26.456534 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-cfb82_f368693e-275e-42a4-8bef-ec9a8d380236/console-operator/3.log" Sep 07 11:41:27.281937 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.281905 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr"] Sep 07 11:41:27.282270 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.282251 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" containerName="gather" Sep 07 11:41:27.282357 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.282274 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" containerName="gather" Sep 07 11:41:27.282357 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.282300 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" containerName="copy" Sep 07 11:41:27.282357 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.282308 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" containerName="copy" Sep 07 11:41:27.282510 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.282375 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" containerName="gather" Sep 07 11:41:27.282510 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.282390 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec6ac6e-d315-4e7d-99d3-0b7577ef3222" containerName="copy" Sep 07 11:41:27.287241 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.287217 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.296075 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.295982 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr"] Sep 07 11:41:27.312670 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.312638 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-2nrt8_24661e4f-4c12-4393-8f27-a47096220e14/volume-data-source-validator/0.log" Sep 07 11:41:27.390939 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.390904 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-sys\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.391176 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.391160 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-lib-modules\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.391391 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.391371 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-podres\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.391603 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.391586 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-proc\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.391743 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.391721 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdj48\" (UniqueName: \"kubernetes.io/projected/338d01be-614e-4a28-ae6b-4870f0df3260-kube-api-access-hdj48\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.492566 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.492520 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-podres\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.492978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.492612 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-proc\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.492978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.492650 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdj48\" (UniqueName: \"kubernetes.io/projected/338d01be-614e-4a28-ae6b-4870f0df3260-kube-api-access-hdj48\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.492978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.492680 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-sys\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.492978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.492690 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-podres\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.492978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.492707 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-lib-modules\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.492978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.492819 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-lib-modules\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.492978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.492873 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-sys\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.492978 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.492914 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/338d01be-614e-4a28-ae6b-4870f0df3260-proc\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.502639 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.502602 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdj48\" (UniqueName: \"kubernetes.io/projected/338d01be-614e-4a28-ae6b-4870f0df3260-kube-api-access-hdj48\") pod \"perf-node-gather-daemonset-vc5vr\" (UID: \"338d01be-614e-4a28-ae6b-4870f0df3260\") " pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.600105 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.600026 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.880943 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.880915 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr"] Sep 07 11:41:27.882790 ip-10-0-138-34 kubenswrapper[2571]: W0907 11:41:27.882765 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod338d01be_614e_4a28_ae6b_4870f0df3260.slice/crio-46e28f81b325180a2f00bf06c61bb7a1d4ec4b9bd45bf4cf9ba1ca0cf8af59df WatchSource:0}: Error finding container 46e28f81b325180a2f00bf06c61bb7a1d4ec4b9bd45bf4cf9ba1ca0cf8af59df: Status 404 returned error can't find the container with id 46e28f81b325180a2f00bf06c61bb7a1d4ec4b9bd45bf4cf9ba1ca0cf8af59df Sep 07 11:41:27.956259 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.956231 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" event={"ID":"338d01be-614e-4a28-ae6b-4870f0df3260","Type":"ContainerStarted","Data":"2dead094613231b5f1d37b89cb60e247e70956a66253268bb547483a7593d0ff"} Sep 07 11:41:27.956378 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.956288 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" event={"ID":"338d01be-614e-4a28-ae6b-4870f0df3260","Type":"ContainerStarted","Data":"46e28f81b325180a2f00bf06c61bb7a1d4ec4b9bd45bf4cf9ba1ca0cf8af59df"} Sep 07 11:41:27.956378 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.956320 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:27.974811 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:27.974760 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" podStartSLOduration=0.974741789 podStartE2EDuration="974.741789ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:41:27.972722414 +0000 UTC m=+2460.447078792" watchObservedRunningTime="2026-09-07 11:41:27.974741789 +0000 UTC m=+2460.449098169" Sep 07 11:41:28.055600 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:28.055574 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-49lhf_cd09d09c-a0d7-4104-8d79-746a7d0b8030/dns/0.log" Sep 07 11:41:28.075633 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:28.075609 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-49lhf_cd09d09c-a0d7-4104-8d79-746a7d0b8030/kube-rbac-proxy/0.log" Sep 07 11:41:28.176773 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:28.176716 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4fw4g_c839fa80-c4fe-4444-b4d3-4de7cef50ec6/dns-node-resolver/0.log" Sep 07 11:41:28.664777 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:28.664751 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r5q7t_973302dc-6097-4017-b889-c7bcd790d603/node-ca/0.log" Sep 07 11:41:29.581081 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:29.581050 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sjcrj_e23c9000-ee54-41ff-8409-5da16eed5f60/serve-healthcheck-canary/0.log" Sep 07 11:41:30.064071 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:30.064039 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zkccb_dc9d7764-52d5-496a-80e5-8802db0a5d5c/kube-rbac-proxy/0.log" Sep 07 11:41:30.082706 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:30.082684 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zkccb_dc9d7764-52d5-496a-80e5-8802db0a5d5c/exporter/0.log" Sep 07 11:41:30.102390 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:30.102348 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zkccb_dc9d7764-52d5-496a-80e5-8802db0a5d5c/extractor/0.log" Sep 07 11:41:33.974078 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:33.974037 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zzq24/perf-node-gather-daemonset-vc5vr" Sep 07 11:41:34.381533 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:34.381502 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-kv4fh_f55c689e-68fb-4fbb-914f-dae8863f36cc/kube-storage-version-migrator-operator/1.log" Sep 07 11:41:34.383189 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:34.383168 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-kv4fh_f55c689e-68fb-4fbb-914f-dae8863f36cc/kube-storage-version-migrator-operator/0.log" Sep 07 11:41:35.434625 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:35.434581 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nv7jl_54fc1277-37e8-4db4-941b-5e6c19a9a3c4/kube-multus-additional-cni-plugins/0.log" Sep 07 11:41:35.454875 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:35.454841 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nv7jl_54fc1277-37e8-4db4-941b-5e6c19a9a3c4/egress-router-binary-copy/0.log" Sep 07 11:41:35.476411 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:35.476379 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nv7jl_54fc1277-37e8-4db4-941b-5e6c19a9a3c4/cni-plugins/0.log" Sep 07 11:41:35.500597 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:35.500570 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nv7jl_54fc1277-37e8-4db4-941b-5e6c19a9a3c4/bond-cni-plugin/0.log" Sep 07 11:41:35.522116 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:35.522085 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nv7jl_54fc1277-37e8-4db4-941b-5e6c19a9a3c4/routeoverride-cni/0.log" Sep 07 11:41:35.543119 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:35.543095 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nv7jl_54fc1277-37e8-4db4-941b-5e6c19a9a3c4/whereabouts-cni-bincopy/0.log" Sep 07 11:41:35.565518 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:35.565496 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nv7jl_54fc1277-37e8-4db4-941b-5e6c19a9a3c4/whereabouts-cni/0.log" Sep 07 11:41:35.777171 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:35.777140 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-m6xpc_17610838-7a01-4354-970a-3aa13a7e64f9/kube-multus/0.log" Sep 07 11:41:35.823896 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:35.823863 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6jc59_cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0/network-metrics-daemon/0.log" Sep 07 11:41:35.840755 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:35.840721 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6jc59_cf3d1c4d-4f6c-40a3-bd80-ee89c163edf0/kube-rbac-proxy/0.log" Sep 07 11:41:37.130512 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:37.130479 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rlg9h_484ea397-1fd4-44c3-8295-9c1833163ccc/ovn-controller/0.log" Sep 07 11:41:37.147790 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:37.147766 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rlg9h_484ea397-1fd4-44c3-8295-9c1833163ccc/ovn-acl-logging/0.log" Sep 07 11:41:37.173218 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:37.173196 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rlg9h_484ea397-1fd4-44c3-8295-9c1833163ccc/kube-rbac-proxy-node/0.log" Sep 07 11:41:37.193697 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:37.193674 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rlg9h_484ea397-1fd4-44c3-8295-9c1833163ccc/kube-rbac-proxy-ovn-metrics/0.log" Sep 07 11:41:37.209197 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:37.209174 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rlg9h_484ea397-1fd4-44c3-8295-9c1833163ccc/northd/0.log" Sep 07 11:41:37.227496 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:37.227477 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rlg9h_484ea397-1fd4-44c3-8295-9c1833163ccc/nbdb/0.log" Sep 07 11:41:37.246217 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:37.246186 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rlg9h_484ea397-1fd4-44c3-8295-9c1833163ccc/sbdb/0.log" Sep 07 11:41:37.302596 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:37.302563 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rlg9h_484ea397-1fd4-44c3-8295-9c1833163ccc/ovnkube-controller/0.log" Sep 07 11:41:38.299577 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:38.299534 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qz5nc_fc08ced6-7a36-4ef2-b8e2-7cd3eb377084/network-check-target-container/0.log" Sep 07 11:41:39.139261 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:39.139230 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zqj2l_9b81c80c-61cb-43ce-ba52-11ba84bc5a12/iptables-alerter/0.log" Sep 07 11:41:39.741776 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:39.741743 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-zh2px_d4897f70-7801-4ebe-a9ff-b1df36797e92/tuned/0.log" Sep 07 11:41:41.377001 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:41.376966 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-zp97j_f3556cb2-da10-4588-bd2d-016629278608/cluster-samples-operator/0.log" Sep 07 11:41:41.394234 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:41.394205 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-zp97j_f3556cb2-da10-4588-bd2d-016629278608/cluster-samples-operator-watch/0.log" Sep 07 11:41:42.250033 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:42.250001 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-859f94c4b-lm4tc_ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d/service-ca-operator/1.log" Sep 07 11:41:42.252259 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:42.252231 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-859f94c4b-lm4tc_ad1e08eb-3b12-4b62-ac0d-7e7c7b470b0d/service-ca-operator/0.log" Sep 07 11:41:42.890587 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:42.890534 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-4qs77_522d0108-c655-4fee-b481-82f8570276b2/csi-driver/0.log" Sep 07 11:41:42.909897 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:42.909868 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-4qs77_522d0108-c655-4fee-b481-82f8570276b2/csi-node-driver-registrar/0.log" Sep 07 11:41:42.929688 ip-10-0-138-34 kubenswrapper[2571]: I0907 11:41:42.929616 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-4qs77_522d0108-c655-4fee-b481-82f8570276b2/csi-liveness-probe/0.log"