Jun 08 11:17:14.805042 ip-10-0-130-94 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jun 08 11:17:14.805055 ip-10-0-130-94 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jun 08 11:17:14.805065 ip-10-0-130-94 systemd[1]: kubelet.service: Failed with result 'resources'. Jun 08 11:17:14.805368 ip-10-0-130-94 systemd[1]: Failed to start Kubernetes Kubelet. Jun 08 11:17:24.923749 ip-10-0-130-94 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jun 08 11:17:24.923767 ip-10-0-130-94 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ca40f30b823441458b38724f6ac80224 -- Jun 08 11:20:10.399921 ip-10-0-130-94 systemd[1]: Starting Kubernetes Kubelet... Jun 08 11:20:10.793253 ip-10-0-130-94 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 08 11:20:10.793253 ip-10-0-130-94 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jun 08 11:20:10.793253 ip-10-0-130-94 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 08 11:20:10.793862 ip-10-0-130-94 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jun 08 11:20:10.793862 ip-10-0-130-94 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 08 11:20:10.794926 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.794842 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jun 08 11:20:10.799801 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799778 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:20:10.799801 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799797 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:20:10.799801 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799802 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:20:10.799801 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799806 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799809 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799813 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799816 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799819 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799822 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799825 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799830 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799833 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799836 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799839 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799842 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799844 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799848 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799851 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799853 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799856 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799861 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799865 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799868 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:20:10.799973 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799873 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799875 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799878 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799881 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799883 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799886 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799889 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799892 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799894 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799897 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799899 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799902 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799905 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799911 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799914 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799917 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799919 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799923 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799927 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799929 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:20:10.800433 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799932 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799934 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799937 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799939 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799944 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799947 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799949 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799952 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799955 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799957 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799960 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799963 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799966 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799970 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799974 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799977 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799980 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799985 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799988 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:20:10.800952 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799991 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799993 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799996 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.799998 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800001 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800003 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800006 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800008 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800011 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800013 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800018 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800021 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800025 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800029 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800033 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800036 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800038 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800041 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800044 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800046 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:20:10.801399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800049 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800051 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800056 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800059 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800062 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800065 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800067 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800070 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800073 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800076 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800079 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800083 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800085 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800088 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800093 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800095 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:20:10.801883 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.800098 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802185 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802193 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802196 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802200 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802203 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802206 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802211 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802214 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802218 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802224 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802227 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802230 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802233 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802236 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802239 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802242 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802246 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802249 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802252 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:20:10.802258 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802256 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802259 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802264 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802267 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802269 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802272 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802275 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802277 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802280 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802284 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802287 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802290 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802292 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802295 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802298 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802303 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802305 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802308 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802311 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:20:10.802762 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802313 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802316 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802319 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802321 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802324 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802326 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802329 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802331 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802334 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802339 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802342 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802344 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802347 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802350 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802353 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802357 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802360 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802362 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802365 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802368 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:20:10.803208 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802373 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802376 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802378 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802381 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802383 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802386 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802388 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802391 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802394 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802396 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802399 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802401 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802404 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802410 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802412 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802415 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802417 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802420 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802422 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802425 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:20:10.803724 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802429 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802432 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802435 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802438 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802444 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802446 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802448 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802452 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802455 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802457 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802460 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802463 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802465 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802468 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802471 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802473 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802476 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802481 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802483 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802486 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:20:10.804197 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.802489 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803111 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803117 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803120 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803123 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803126 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803129 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803131 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803134 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803140 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803142 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803145 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803148 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803151 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803154 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803156 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803159 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803162 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803165 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803168 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:20:10.804681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803171 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803174 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803180 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803182 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803185 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803188 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803190 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803193 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803196 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803199 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803201 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803204 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803206 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803209 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803212 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803217 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803220 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803223 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803226 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803228 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:20:10.805167 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803231 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803233 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803236 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803241 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803245 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803249 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803251 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803257 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803260 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803263 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803265 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803269 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803276 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803279 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803281 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803284 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803287 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803290 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803292 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:20:10.805672 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803295 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803302 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803305 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803308 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803311 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803313 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803316 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803319 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803321 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803324 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803327 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803330 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803334 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803337 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803340 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803342 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803345 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803348 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803351 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803353 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:20:10.806118 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803356 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803359 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803361 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803364 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803367 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803372 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803375 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803378 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803381 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803384 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803387 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803391 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803394 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803398 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803401 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803403 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803409 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803411 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803415 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:20:10.806622 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803419 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.803422 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803565 2569 flags.go:64] FLAG: --address="0.0.0.0" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803685 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803694 2569 flags.go:64] FLAG: --anonymous-auth="true" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803699 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803703 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803706 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803710 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803722 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803725 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803728 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803732 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803735 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803738 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803741 2569 flags.go:64] FLAG: --cgroup-root="" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803744 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803747 2569 flags.go:64] FLAG: --client-ca-file="" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803750 2569 flags.go:64] FLAG: --cloud-provider="external" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803753 2569 flags.go:64] FLAG: --cluster-dns="[]" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803759 2569 flags.go:64] FLAG: --cluster-domain="" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803762 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803765 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803768 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jun 08 11:20:10.807071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803771 2569 flags.go:64] FLAG: --container-log-max-files="5" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803775 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803777 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803780 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803784 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803787 2569 flags.go:64] FLAG: --contention-profiling="false" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803790 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803792 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803795 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803798 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803802 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803805 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803808 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803811 2569 flags.go:64] FLAG: --enable-load-reader="false" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803816 2569 flags.go:64] FLAG: --enable-server="true" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803819 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803823 2569 flags.go:64] FLAG: --event-burst="100" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803828 2569 flags.go:64] FLAG: --event-qps="50" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803833 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803836 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803840 2569 flags.go:64] FLAG: --eviction-hard="" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803844 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803849 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803853 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803856 2569 flags.go:64] FLAG: --eviction-soft="" Jun 08 11:20:10.807642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803861 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803865 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803870 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803873 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803877 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803881 2569 flags.go:64] FLAG: --fail-swap-on="true" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803884 2569 flags.go:64] FLAG: --feature-gates="" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803889 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803892 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803895 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803898 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803901 2569 flags.go:64] FLAG: --healthz-port="10248" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803904 2569 flags.go:64] FLAG: --help="false" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803906 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-130-94.ec2.internal" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803909 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803913 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803916 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803919 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803922 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803925 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803928 2569 flags.go:64] FLAG: --image-service-endpoint="" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803931 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803933 2569 flags.go:64] FLAG: --kube-api-burst="100" Jun 08 11:20:10.808226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803936 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803939 2569 flags.go:64] FLAG: --kube-api-qps="50" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803942 2569 flags.go:64] FLAG: --kube-reserved="" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803944 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803947 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803950 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803953 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803955 2569 flags.go:64] FLAG: --lock-file="" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803958 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803960 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803963 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803969 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803973 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803975 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803978 2569 flags.go:64] FLAG: --logging-format="text" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803981 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803984 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803987 2569 flags.go:64] FLAG: --manifest-url="" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803990 2569 flags.go:64] FLAG: --manifest-url-header="" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803994 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.803997 2569 flags.go:64] FLAG: --max-open-files="1000000" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804000 2569 flags.go:64] FLAG: --max-pods="110" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804003 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804006 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804012 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jun 08 11:20:10.808888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804015 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804019 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804022 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804025 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804037 2569 flags.go:64] FLAG: --node-status-max-images="50" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804040 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804043 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804046 2569 flags.go:64] FLAG: --pod-cidr="" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804049 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746976bfb3a92b93dacc79c151b98fd4c932eef19c3e2c1516b14eebcac9184d" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804054 2569 flags.go:64] FLAG: --pod-manifest-path="" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804057 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804060 2569 flags.go:64] FLAG: --pods-per-core="0" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804063 2569 flags.go:64] FLAG: --port="10250" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804066 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804069 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b6e3cf03e8a02aa3" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804072 2569 flags.go:64] FLAG: --qos-reserved="" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804075 2569 flags.go:64] FLAG: --read-only-port="10255" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804078 2569 flags.go:64] FLAG: --register-node="true" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804080 2569 flags.go:64] FLAG: --register-with-taints="" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804084 2569 flags.go:64] FLAG: --registry-burst="10" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804087 2569 flags.go:64] FLAG: --registry-qps="5" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804090 2569 flags.go:64] FLAG: --reserved-cpus="" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804093 2569 flags.go:64] FLAG: --reserved-memory="" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804096 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804099 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jun 08 11:20:10.809485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804102 2569 flags.go:64] FLAG: --rotate-certificates="false" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804105 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804108 2569 flags.go:64] FLAG: --runonce="false" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804110 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804113 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804116 2569 flags.go:64] FLAG: --seccomp-default="false" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804120 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804123 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804126 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804132 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804135 2569 flags.go:64] FLAG: --storage-driver-password="root" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804137 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804140 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804143 2569 flags.go:64] FLAG: --storage-driver-user="root" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804146 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804149 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804151 2569 flags.go:64] FLAG: --system-cgroups="" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804154 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804161 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804163 2569 flags.go:64] FLAG: --tls-cert-file="" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804166 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804170 2569 flags.go:64] FLAG: --tls-min-version="" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804173 2569 flags.go:64] FLAG: --tls-private-key-file="" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804175 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804178 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jun 08 11:20:10.810130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804181 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804184 2569 flags.go:64] FLAG: --v="2" Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804188 2569 flags.go:64] FLAG: --version="false" Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804192 2569 flags.go:64] FLAG: --vmodule="" Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804196 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.804199 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804316 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804320 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804324 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804327 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804330 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804333 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804336 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804340 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804342 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804345 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804349 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804352 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804354 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804357 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804360 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:20:10.810731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804363 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804366 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804368 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804373 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804377 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804380 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804383 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804385 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804388 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804391 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804393 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804396 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804398 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804401 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804403 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804407 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804409 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804413 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804416 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:20:10.811238 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804419 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804422 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804425 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804427 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804430 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804432 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804435 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804438 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804441 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804444 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804447 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804449 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804452 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804455 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804457 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804460 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804462 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804465 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804467 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804470 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:20:10.811709 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804472 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804475 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804477 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804479 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804482 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804484 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804487 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804490 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804492 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804496 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804499 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804501 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804504 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804506 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804509 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804512 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804514 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804517 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804519 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:20:10.812183 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804522 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804526 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804528 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804531 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804534 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804536 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804539 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804542 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804544 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804548 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804551 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804554 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804557 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804559 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804562 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804564 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804567 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804569 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804572 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804574 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:20:10.812675 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804577 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804579 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804581 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804585 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804587 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.804589 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.805304 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.812035 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.812050 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.812069 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.812078 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812132 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812137 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812140 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812144 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:20:10.813151 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812147 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812151 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812155 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812157 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812160 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812163 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812166 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812168 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812171 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812173 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812175 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812178 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812181 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812183 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812186 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812188 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812191 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812194 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812197 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:20:10.813517 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812200 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812203 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812205 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812208 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812210 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812213 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812216 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812218 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812221 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812223 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812226 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812228 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812231 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812233 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812236 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812238 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812240 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812243 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812245 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812248 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812250 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:20:10.813982 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812252 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812255 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812257 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812259 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812262 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812264 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812267 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812269 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812271 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812274 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812276 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812278 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812281 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812284 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812287 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812289 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812292 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812297 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812300 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:20:10.814464 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812303 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812306 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812308 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812311 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812314 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812316 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812319 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812321 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812323 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812326 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812328 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812331 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812333 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812335 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812338 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812340 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812343 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812345 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812347 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812349 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:20:10.814927 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812352 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812354 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812357 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812359 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812361 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812365 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812368 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812372 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812375 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812377 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812381 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812383 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812386 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812389 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812392 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:20:10.815416 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812394 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.812399 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812507 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812511 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812514 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812517 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812520 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812522 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812525 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812527 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812530 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812532 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812535 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812538 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:20:10.815787 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812541 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812546 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812549 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812552 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812555 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812558 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812561 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812564 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812566 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812569 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812571 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812574 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812576 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812579 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812582 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812585 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812587 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812589 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812592 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812595 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:20:10.816116 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812597 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812599 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812602 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812604 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812623 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812627 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812630 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812633 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812635 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812638 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812640 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812643 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812645 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812647 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812650 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812652 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812654 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812657 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812659 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:20:10.816580 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812662 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812664 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812667 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812669 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812671 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812674 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812677 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812680 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812682 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812685 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812688 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812691 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812693 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812696 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812699 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812701 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812704 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812706 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812709 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812711 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:20:10.817068 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812713 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812716 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812718 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812721 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812723 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812726 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812728 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812730 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812733 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812735 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812738 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812740 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812743 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812745 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812748 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812751 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812753 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812756 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812759 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812762 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:20:10.817532 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812764 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812766 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812769 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812771 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812774 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812776 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812778 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:10.812781 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.812786 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.812885 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.814790 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.816071 2569 server.go:1019] "Starting client certificate rotation" Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.816172 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 08 11:20:10.818022 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.816206 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 08 11:20:10.818359 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.818344 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 08 11:20:10.819947 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.819930 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 08 11:20:10.829938 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.829921 2569 log.go:25] "Validated CRI v1 runtime API" Jun 08 11:20:10.832276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.832263 2569 log.go:25] "Validated CRI v1 image API" Jun 08 11:20:10.833825 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.833810 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jun 08 11:20:10.836863 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.836844 2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b019d85a-65ec-4daf-b19b-34133d3c9c1b:/dev/nvme0n1p4 bf68f1fa-90fd-407b-ad1d-816c3661e6c0:/dev/nvme0n1p3] Jun 08 11:20:10.836918 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.836863 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jun 08 11:20:10.843639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.843515 2569 manager.go:216] Machine: {Timestamp:2026-06-08 11:20:10.842238494 +0000 UTC m=+0.326637348 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3211842 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29abc798ab5805afb32116d3b35ac0 SystemUUID:ec29abc7-98ab-5805-afb3-2116d3b35ac0 BootID:ca40f30b-8234-4145-8b38-724f6ac80224 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0b:8a:d7:ad:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0b:8a:d7:ad:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:81:a8:8f:ee:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jun 08 11:20:10.843639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.843631 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jun 08 11:20:10.843766 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.843713 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.118.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260602-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jun 08 11:20:10.846735 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.846712 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jun 08 11:20:10.846859 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.846739 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jun 08 11:20:10.848228 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.848215 2569 topology_manager.go:138] "Creating topology manager with none policy" Jun 08 11:20:10.848228 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.848228 2569 container_manager_linux.go:309] "Creating device plugin manager" Jun 08 11:20:10.848313 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.848236 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 08 11:20:10.849012 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.849001 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 08 11:20:10.849831 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.849822 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jun 08 11:20:10.852708 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.852694 2569 state_mem.go:36] "Initialized new in-memory state store" Jun 08 11:20:10.852708 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.852701 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 08 11:20:10.853009 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.852998 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jun 08 11:20:10.853052 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.853031 2569 kubelet.go:486] "Attempting to sync node with API server" Jun 08 11:20:10.853052 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.853049 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jun 08 11:20:10.853102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.853063 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jun 08 11:20:10.853102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.853073 2569 kubelet.go:398] "Adding apiserver pod source" Jun 08 11:20:10.853102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.853080 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jun 08 11:20:10.853437 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.853421 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 08 11:20:10.853488 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.853450 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 08 11:20:10.856062 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.856047 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jun 08 11:20:10.858028 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.858016 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jun 08 11:20:10.858063 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.858045 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jun 08 11:20:10.859184 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859175 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jun 08 11:20:10.859221 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859190 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jun 08 11:20:10.859221 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859196 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jun 08 11:20:10.859221 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859202 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jun 08 11:20:10.859221 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859211 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jun 08 11:20:10.859221 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859217 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jun 08 11:20:10.859345 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859225 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jun 08 11:20:10.859345 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859230 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jun 08 11:20:10.859345 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859236 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jun 08 11:20:10.859345 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859241 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jun 08 11:20:10.859345 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859249 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jun 08 11:20:10.859345 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859267 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jun 08 11:20:10.859949 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859936 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jun 08 11:20:10.859949 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.859948 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jun 08 11:20:10.863170 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.863145 2569 server.go:1268] "Started kubelet" Jun 08 11:20:10.863255 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.863224 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jun 08 11:20:10.863960 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.863418 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jun 08 11:20:10.863960 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.863846 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jun 08 11:20:10.864130 ip-10-0-130-94 systemd[1]: Started Kubernetes Kubelet. Jun 08 11:20:10.866024 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:10.865985 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jun 08 11:20:10.866088 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.866023 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jun 08 11:20:10.866088 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:10.866029 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jun 08 11:20:10.866265 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.866249 2569 server.go:310] "Adding debug handlers to kubelet server" Jun 08 11:20:10.867586 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.867557 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jun 08 11:20:10.868703 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:10.867576 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-94.ec2.internal.18b71721a1bb96a7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-94.ec2.internal,UID:ip-10-0-130-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-94.ec2.internal,},FirstTimestamp:2026-06-08 11:20:10.863105703 +0000 UTC m=+0.347504557,LastTimestamp:2026-06-08 11:20:10.863105703 +0000 UTC m=+0.347504557,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-94.ec2.internal,}" Jun 08 11:20:10.872292 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.872276 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jun 08 11:20:10.872370 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.872307 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jun 08 11:20:10.872679 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:10.872656 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:10.873140 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.873039 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jun 08 11:20:10.873206 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.873145 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jun 08 11:20:10.873206 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.873150 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jun 08 11:20:10.873402 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.873348 2569 reconstruct.go:97] "Volume reconstruction finished" Jun 08 11:20:10.873402 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.873362 2569 reconciler.go:29] "Reconciler: start to sync state" Jun 08 11:20:10.873955 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:10.873930 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jun 08 11:20:10.874062 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.873967 2569 factory.go:55] Registering systemd factory Jun 08 11:20:10.874148 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.874136 2569 factory.go:223] Registration of the systemd container factory successfully Jun 08 11:20:10.874419 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.874409 2569 factory.go:153] Registering CRI-O factory Jun 08 11:20:10.874513 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.874503 2569 factory.go:223] Registration of the crio container factory successfully Jun 08 11:20:10.874663 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.874652 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jun 08 11:20:10.874779 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.874763 2569 factory.go:103] Registering Raw factory Jun 08 11:20:10.874868 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.874791 2569 manager.go:1196] Started watching for new ooms in manager Jun 08 11:20:10.874991 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:10.874965 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jun 08 11:20:10.875205 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.875189 2569 manager.go:349] Starting recovery of all containers Jun 08 11:20:10.883045 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.883024 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5cbc" Jun 08 11:20:10.887511 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:10.887492 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jun 08 11:20:10.887755 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.887742 2569 manager.go:354] Recovery completed Jun 08 11:20:10.889464 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.889443 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5cbc" Jun 08 11:20:10.892134 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.892123 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:20:10.894261 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.894245 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:20:10.894331 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.894272 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:20:10.894331 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.894283 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:20:10.894794 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.894777 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jun 08 11:20:10.894794 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.894793 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jun 08 11:20:10.894890 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.894806 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:20:10.894890 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.894829 2569 state_mem.go:36] "Initialized new in-memory state store" Jun 08 11:20:10.895523 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.895508 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:20:10.895593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.895539 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:20:10.895593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.895552 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:20:10.898685 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.898673 2569 policy_none.go:49] "None policy: Start" Jun 08 11:20:10.898685 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.898688 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jun 08 11:20:10.898776 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.898697 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jun 08 11:20:10.899771 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.899762 2569 policy_none.go:47] "Start" Jun 08 11:20:10.945679 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.935960 2569 manager.go:337] "Starting Device Plugin manager" Jun 08 11:20:10.945679 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:10.935987 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jun 08 11:20:10.945679 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.935997 2569 server.go:87] "Starting device plugin registration server" Jun 08 11:20:10.945679 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.936223 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jun 08 11:20:10.945679 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.936236 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jun 08 11:20:10.945679 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.936326 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jun 08 11:20:10.945679 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.936407 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jun 08 11:20:10.945679 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.936427 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jun 08 11:20:10.945679 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:10.936998 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jun 08 11:20:10.945679 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:10.937053 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:10.946127 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.946005 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jun 08 11:20:10.947517 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.947496 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jun 08 11:20:10.947630 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.947545 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jun 08 11:20:10.947630 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.947569 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jun 08 11:20:10.947630 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:10.947605 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jun 08 11:20:10.949625 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:10.949589 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 08 11:20:11.037108 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.037084 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:20:11.038820 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.038798 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:20:11.038934 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.038836 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:20:11.038934 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.038851 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:20:11.038934 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.038882 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.047723 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.047665 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.047723 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.047693 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-94.ec2.internal\": node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:11.047875 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.047798 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal"] Jun 08 11:20:11.047875 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.047864 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:20:11.049197 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.049184 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:20:11.049291 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.049224 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:20:11.049291 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.049239 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:20:11.050437 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.050422 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:20:11.050564 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.050548 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.050633 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.050583 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:20:11.051146 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.051128 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:20:11.051207 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.051157 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:20:11.051207 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.051167 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:20:11.052529 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.052511 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:20:11.052602 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.052553 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:20:11.052602 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.052563 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:20:11.053710 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.053694 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.053774 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.053724 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:20:11.054396 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.054382 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:20:11.054451 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.054406 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:20:11.054451 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.054416 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:20:11.066803 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.066789 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:11.072144 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.072129 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-94.ec2.internal\" not found" node="ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.074513 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.074420 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d77966ae3dbd03711bb241eda3758fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"d77966ae3dbd03711bb241eda3758fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.074639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.074594 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d77966ae3dbd03711bb241eda3758fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"d77966ae3dbd03711bb241eda3758fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.074700 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.074667 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b69eb2fa1fe4d666208120158aec889b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-94.ec2.internal\" (UID: \"b69eb2fa1fe4d666208120158aec889b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.077559 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.077543 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-94.ec2.internal\" not found" node="ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.167338 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.167308 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:11.175143 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.175114 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d77966ae3dbd03711bb241eda3758fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"d77966ae3dbd03711bb241eda3758fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.175269 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.175149 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d77966ae3dbd03711bb241eda3758fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"d77966ae3dbd03711bb241eda3758fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.175269 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.175165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b69eb2fa1fe4d666208120158aec889b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-94.ec2.internal\" (UID: \"b69eb2fa1fe4d666208120158aec889b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.175269 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.175213 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b69eb2fa1fe4d666208120158aec889b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-94.ec2.internal\" (UID: \"b69eb2fa1fe4d666208120158aec889b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.175269 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.175215 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d77966ae3dbd03711bb241eda3758fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"d77966ae3dbd03711bb241eda3758fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.175269 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.175214 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d77966ae3dbd03711bb241eda3758fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal\" (UID: \"d77966ae3dbd03711bb241eda3758fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.268333 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.268293 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:11.369095 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.369020 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:11.376339 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.376316 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.382054 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.382035 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jun 08 11:20:11.469458 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.469413 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:11.569926 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.569888 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:11.670535 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.670452 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:11.770939 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.770903 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:11.815509 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.815480 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jun 08 11:20:11.816154 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.815671 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jun 08 11:20:11.847103 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:11.847071 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77966ae3dbd03711bb241eda3758fb8.slice/crio-ec0b6cdd3e941e756136f1160ed588b7957c61f899a77a73b5c1669d3d975fe4 WatchSource:0}: Error finding container ec0b6cdd3e941e756136f1160ed588b7957c61f899a77a73b5c1669d3d975fe4: Status 404 returned error can't find the container with id ec0b6cdd3e941e756136f1160ed588b7957c61f899a77a73b5c1669d3d975fe4 Jun 08 11:20:11.847299 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:11.847284 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69eb2fa1fe4d666208120158aec889b.slice/crio-ed208578d8d011602d4ab33e520a9e4f2bf5d4aea51ebe387a98d294cf263fe5 WatchSource:0}: Error finding container ed208578d8d011602d4ab33e520a9e4f2bf5d4aea51ebe387a98d294cf263fe5: Status 404 returned error can't find the container with id ed208578d8d011602d4ab33e520a9e4f2bf5d4aea51ebe387a98d294cf263fe5 Jun 08 11:20:11.851927 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.851909 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 08 11:20:11.871759 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.871736 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:11.892366 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.892333 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-07 11:15:10 +0000 UTC" deadline="2027-11-21 13:41:49.395140925 +0000 UTC" Jun 08 11:20:11.892366 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.892362 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12746h21m37.502784046s" Jun 08 11:20:11.950898 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.950808 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" event={"ID":"d77966ae3dbd03711bb241eda3758fb8","Type":"ContainerStarted","Data":"ec0b6cdd3e941e756136f1160ed588b7957c61f899a77a73b5c1669d3d975fe4"} Jun 08 11:20:11.951922 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:11.951903 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" event={"ID":"b69eb2fa1fe4d666208120158aec889b","Type":"ContainerStarted","Data":"ed208578d8d011602d4ab33e520a9e4f2bf5d4aea51ebe387a98d294cf263fe5"} Jun 08 11:20:11.972054 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:11.972036 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:12.072578 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.072541 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:12.076944 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.076929 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jun 08 11:20:12.173138 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.173101 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:12.176789 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.176772 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jun 08 11:20:12.274048 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.273969 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-94.ec2.internal\" not found" Jun 08 11:20:12.278763 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.278741 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jun 08 11:20:12.373116 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.373086 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" Jun 08 11:20:12.386084 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.386057 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 08 11:20:12.386978 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.386957 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" Jun 08 11:20:12.395377 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.395355 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 08 11:20:12.854944 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.854912 2569 apiserver.go:52] "Watching apiserver" Jun 08 11:20:12.862196 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.862171 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jun 08 11:20:12.862652 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.862631 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-bx9jt","kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal","openshift-dns/node-resolver-bkxdh","openshift-image-registry/node-ca-zmxqf","openshift-multus/multus-additional-cni-plugins-6qdwq","openshift-multus/multus-hg6x5","openshift-multus/network-metrics-daemon-kthbk","openshift-network-diagnostics/network-check-target-m6fn6","openshift-network-operator/iptables-alerter-k2t8t","kube-system/global-pull-secret-syncer-dvnth","kube-system/konnectivity-agent-hjb7c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd","openshift-cluster-node-tuning-operator/tuned-2m76c"] Jun 08 11:20:12.864185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.864163 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.865534 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.865511 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.866470 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.866447 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jun 08 11:20:12.866602 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.866582 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jun 08 11:20:12.866730 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.866713 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.866811 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.866793 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jun 08 11:20:12.866921 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.866902 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hjhk9\"" Jun 08 11:20:12.866981 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.866795 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jun 08 11:20:12.867727 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.867706 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6jwrr\"" Jun 08 11:20:12.867824 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.867714 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jun 08 11:20:12.867824 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.867735 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jun 08 11:20:12.867996 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.867979 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bkxdh" Jun 08 11:20:12.868067 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.868058 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jun 08 11:20:12.870058 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.870039 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jun 08 11:20:12.870357 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.870336 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6b64w\"" Jun 08 11:20:12.870458 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.870406 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jun 08 11:20:12.870458 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.870427 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-987fw\"" Jun 08 11:20:12.871503 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.870789 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jun 08 11:20:12.871503 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.870865 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zmxqf" Jun 08 11:20:12.871503 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.871322 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jun 08 11:20:12.871503 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.871475 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.872520 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.872501 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:12.872726 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.872586 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:12.872848 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.872828 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jun 08 11:20:12.873024 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.873003 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jun 08 11:20:12.873754 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.873734 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jun 08 11:20:12.873840 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.873762 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jun 08 11:20:12.874194 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.874091 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jun 08 11:20:12.874194 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.874130 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xwd87\"" Jun 08 11:20:12.874194 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.874138 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qgrtz\"" Jun 08 11:20:12.874194 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.874162 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jun 08 11:20:12.876171 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.874819 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:12.876171 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.874891 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:12.876522 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.876183 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.877478 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.877461 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k2t8t" Jun 08 11:20:12.878816 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.878716 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jun 08 11:20:12.878902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.878820 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jun 08 11:20:12.878902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.878892 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jun 08 11:20:12.879011 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.878951 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jun 08 11:20:12.879011 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.878993 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jun 08 11:20:12.879011 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.879004 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jun 08 11:20:12.879147 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.879095 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v5mxj\"" Jun 08 11:20:12.879244 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.879228 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:12.879299 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.879281 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:12.879497 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.879483 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jun 08 11:20:12.879559 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.879543 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7btmj\"" Jun 08 11:20:12.879793 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.879778 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jun 08 11:20:12.880201 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.880184 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:12.880276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.880188 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jun 08 11:20:12.882419 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.882389 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jun 08 11:20:12.882419 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.882389 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m59jk\"" Jun 08 11:20:12.882552 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.882441 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jun 08 11:20:12.885358 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885337 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-sysctl-conf\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.885454 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885381 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-systemd\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.885454 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885408 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-run\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.885454 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885429 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-socket-dir-parent\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.885580 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885453 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbn7\" (UniqueName: \"kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7\") pod \"network-check-target-m6fn6\" (UID: \"f8064e28-3a81-423c-b1b4-3610c44f0f46\") " pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:12.885580 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885482 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-hostroot\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.885580 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885519 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-sysctl-d\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.885580 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885539 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-sys\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.885580 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885559 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-os-release\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.885756 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885588 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-var-lib-cni-bin\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.885756 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885635 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-conf-dir\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.885756 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885652 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-daemon-config\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.885756 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885680 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-etc-kubernetes\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.885756 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885698 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/80d9b28a-6045-49bf-8f48-304a8396799f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.885756 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885713 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-sys-fs\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.885756 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885727 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-modprobe-d\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.885756 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885752 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-var-lib-kubelet\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885767 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-run-netns\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885780 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-system-cni-dir\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885800 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68kmp\" (UniqueName: \"kubernetes.io/projected/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-kube-api-access-68kmp\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885815 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-device-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885834 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-kubernetes\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885852 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-lib-modules\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885884 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/80d9b28a-6045-49bf-8f48-304a8396799f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79hjp\" (UniqueName: \"kubernetes.io/projected/25861972-bb71-4dc8-bfae-31d8c5a5bd8e-kube-api-access-79hjp\") pod \"node-ca-zmxqf\" (UID: \"25861972-bb71-4dc8-bfae-31d8c5a5bd8e\") " pod="openshift-image-registry/node-ca-zmxqf" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-os-release\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.885993 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-registration-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886015 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-cnibin\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.886068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqzzp\" (UniqueName: \"kubernetes.io/projected/80d9b28a-6045-49bf-8f48-304a8396799f-kube-api-access-hqzzp\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886084 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-run-k8s-cni-cncf-io\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886113 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-run-multus-certs\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886134 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kklm5\" (UniqueName: \"kubernetes.io/projected/be0f4959-65ba-40a2-aca1-76b571c2ce30-kube-api-access-kklm5\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886163 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-host\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886184 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-cni-dir\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886210 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be0f4959-65ba-40a2-aca1-76b571c2ce30-cni-binary-copy\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886226 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-var-lib-cni-multus\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886239 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25861972-bb71-4dc8-bfae-31d8c5a5bd8e-host\") pod \"node-ca-zmxqf\" (UID: \"25861972-bb71-4dc8-bfae-31d8c5a5bd8e\") " pod="openshift-image-registry/node-ca-zmxqf" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886255 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80d9b28a-6045-49bf-8f48-304a8396799f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886270 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/722e256c-433a-4b5f-aba9-960dd03efc28-tmp\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886305 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-cnibin\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886348 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/722e256c-433a-4b5f-aba9-960dd03efc28-etc-tuned\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886382 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822bda63-ed87-40ff-86a1-5ebee377ed5a-tmp-dir\") pod \"node-resolver-bkxdh\" (UID: \"822bda63-ed87-40ff-86a1-5ebee377ed5a\") " pod="openshift-dns/node-resolver-bkxdh" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/25861972-bb71-4dc8-bfae-31d8c5a5bd8e-serviceca\") pod \"node-ca-zmxqf\" (UID: \"25861972-bb71-4dc8-bfae-31d8c5a5bd8e\") " pod="openshift-image-registry/node-ca-zmxqf" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886430 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-kubelet-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.886650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886453 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-sysconfig\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.887286 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886474 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqn5l\" (UniqueName: \"kubernetes.io/projected/722e256c-433a-4b5f-aba9-960dd03efc28-kube-api-access-xqn5l\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.887286 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-system-cni-dir\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.887286 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886522 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-etc-selinux\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.887286 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886546 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brhqt\" (UniqueName: \"kubernetes.io/projected/08095808-76be-4f3c-8ff2-c7dfaeb5c832-kube-api-access-brhqt\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.887286 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886568 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-var-lib-kubelet\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.887286 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886591 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/822bda63-ed87-40ff-86a1-5ebee377ed5a-hosts-file\") pod \"node-resolver-bkxdh\" (UID: \"822bda63-ed87-40ff-86a1-5ebee377ed5a\") " pod="openshift-dns/node-resolver-bkxdh" Jun 08 11:20:12.887286 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886638 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-socket-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.887286 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.886662 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvtn\" (UniqueName: \"kubernetes.io/projected/822bda63-ed87-40ff-86a1-5ebee377ed5a-kube-api-access-9vvtn\") pod \"node-resolver-bkxdh\" (UID: \"822bda63-ed87-40ff-86a1-5ebee377ed5a\") " pod="openshift-dns/node-resolver-bkxdh" Jun 08 11:20:12.889554 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.889528 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 08 11:20:12.896222 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.896201 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 08 11:20:12.911472 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.911451 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5clmh" Jun 08 11:20:12.917522 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.917500 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5clmh" Jun 08 11:20:12.974696 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.974661 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jun 08 11:20:12.987677 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987645 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-socket-dir-parent\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.987865 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987689 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-node-log\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.987865 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987715 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbn7\" (UniqueName: \"kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7\") pod \"network-check-target-m6fn6\" (UID: \"f8064e28-3a81-423c-b1b4-3610c44f0f46\") " pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:12.987865 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987783 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-socket-dir-parent\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.987865 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987827 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-hostroot\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.987865 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987851 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-run-systemd\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.987865 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987865 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6534a1c3-5859-4fc5-8b05-ca877b16aa77-ovnkube-script-lib\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987882 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-sysctl-d\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987933 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-sys\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987970 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-os-release\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987975 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-sys\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987999 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-var-lib-cni-bin\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.987937 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-hostroot\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988004 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-sysctl-d\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988025 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-conf-dir\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988048 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-daemon-config\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988048 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-var-lib-cni-bin\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988069 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-conf-dir\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988073 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-etc-kubernetes\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988090 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-os-release\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988110 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-etc-kubernetes\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988117 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.988145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988139 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/80d9b28a-6045-49bf-8f48-304a8396799f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988156 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-sys-fs\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988176 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-modprobe-d\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-var-lib-kubelet\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-run-netns\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988234 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-sys-fs\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988243 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6534a1c3-5859-4fc5-8b05-ca877b16aa77-ovnkube-config\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988266 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-system-cni-dir\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988289 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68kmp\" (UniqueName: \"kubernetes.io/projected/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-kube-api-access-68kmp\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988295 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-var-lib-kubelet\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988311 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-device-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988317 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-modprobe-d\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988333 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-kubernetes\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988351 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-run-netns\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988359 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-lib-modules\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988381 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-systemd-units\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988396 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-system-cni-dir\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.988860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988406 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/80d9b28a-6045-49bf-8f48-304a8396799f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988441 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-kubernetes\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988448 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988471 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-run-netns\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988489 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-device-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988494 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-run-ovn\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79hjp\" (UniqueName: \"kubernetes.io/projected/25861972-bb71-4dc8-bfae-31d8c5a5bd8e-kube-api-access-79hjp\") pod \"node-ca-zmxqf\" (UID: \"25861972-bb71-4dc8-bfae-31d8c5a5bd8e\") " pod="openshift-image-registry/node-ca-zmxqf" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988541 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-os-release\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988570 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-registration-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-cnibin\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-var-lib-openvswitch\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.988639 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988660 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqzzp\" (UniqueName: \"kubernetes.io/projected/80d9b28a-6045-49bf-8f48-304a8396799f-kube-api-access-hqzzp\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988700 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-daemon-config\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.988720 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs podName:2413fc4e-6ee1-47a8-8621-657e43ccfcf4 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:13.488683185 +0000 UTC m=+2.973082056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs") pod "network-metrics-daemon-kthbk" (UID: "2413fc4e-6ee1-47a8-8621-657e43ccfcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988728 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/80d9b28a-6045-49bf-8f48-304a8396799f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-run-k8s-cni-cncf-io\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.989623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988771 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-run-multus-certs\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988798 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-kubelet\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988805 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-os-release\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-slash\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988848 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0219d35-4a8d-4383-acda-cc5fad912ade-konnectivity-ca\") pod \"konnectivity-agent-hjb7c\" (UID: \"b0219d35-4a8d-4383-acda-cc5fad912ade\") " pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988875 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kklm5\" (UniqueName: \"kubernetes.io/projected/be0f4959-65ba-40a2-aca1-76b571c2ce30-kube-api-access-kklm5\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988914 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-lib-modules\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988930 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-cnibin\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988938 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/80d9b28a-6045-49bf-8f48-304a8396799f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988963 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-run-multus-certs\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-host\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989028 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-cni-dir\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989052 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be0f4959-65ba-40a2-aca1-76b571c2ce30-cni-binary-copy\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989073 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-var-lib-cni-multus\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989108 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989113 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-multus-cni-dir\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.990321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989108 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/74c2edb1-405d-4d3d-ae3f-d07b87d495f9-iptables-alerter-script\") pod \"iptables-alerter-k2t8t\" (UID: \"74c2edb1-405d-4d3d-ae3f-d07b87d495f9\") " pod="openshift-network-operator/iptables-alerter-k2t8t" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.988977 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-registration-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989074 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-host\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989005 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-run-k8s-cni-cncf-io\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989160 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-var-lib-cni-multus\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25861972-bb71-4dc8-bfae-31d8c5a5bd8e-host\") pod \"node-ca-zmxqf\" (UID: \"25861972-bb71-4dc8-bfae-31d8c5a5bd8e\") " pod="openshift-image-registry/node-ca-zmxqf" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80d9b28a-6045-49bf-8f48-304a8396799f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/722e256c-433a-4b5f-aba9-960dd03efc28-tmp\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989273 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25861972-bb71-4dc8-bfae-31d8c5a5bd8e-host\") pod \"node-ca-zmxqf\" (UID: \"25861972-bb71-4dc8-bfae-31d8c5a5bd8e\") " pod="openshift-image-registry/node-ca-zmxqf" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989314 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-dbus\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989342 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6534a1c3-5859-4fc5-8b05-ca877b16aa77-env-overrides\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74c2edb1-405d-4d3d-ae3f-d07b87d495f9-host-slash\") pod \"iptables-alerter-k2t8t\" (UID: \"74c2edb1-405d-4d3d-ae3f-d07b87d495f9\") " pod="openshift-network-operator/iptables-alerter-k2t8t" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989410 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-cnibin\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989437 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/722e256c-433a-4b5f-aba9-960dd03efc28-etc-tuned\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989460 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822bda63-ed87-40ff-86a1-5ebee377ed5a-tmp-dir\") pod \"node-resolver-bkxdh\" (UID: \"822bda63-ed87-40ff-86a1-5ebee377ed5a\") " pod="openshift-dns/node-resolver-bkxdh" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989483 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6534a1c3-5859-4fc5-8b05-ca877b16aa77-ovn-node-metrics-cert\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/25861972-bb71-4dc8-bfae-31d8c5a5bd8e-serviceca\") pod \"node-ca-zmxqf\" (UID: \"25861972-bb71-4dc8-bfae-31d8c5a5bd8e\") " pod="openshift-image-registry/node-ca-zmxqf" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989500 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jun 08 11:20:12.990937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989516 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80d9b28a-6045-49bf-8f48-304a8396799f-cnibin\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989540 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-kubelet-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989570 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-kubelet-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989584 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-sysconfig\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqn5l\" (UniqueName: \"kubernetes.io/projected/722e256c-433a-4b5f-aba9-960dd03efc28-kube-api-access-xqn5l\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989653 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-system-cni-dir\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989675 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-kubelet-config\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989698 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6ld\" (UniqueName: \"kubernetes.io/projected/6534a1c3-5859-4fc5-8b05-ca877b16aa77-kube-api-access-9m6ld\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-etc-selinux\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989747 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80d9b28a-6045-49bf-8f48-304a8396799f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989788 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822bda63-ed87-40ff-86a1-5ebee377ed5a-tmp-dir\") pod \"node-resolver-bkxdh\" (UID: \"822bda63-ed87-40ff-86a1-5ebee377ed5a\") " pod="openshift-dns/node-resolver-bkxdh" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989814 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-sysconfig\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989843 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-run-openvswitch\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989858 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-etc-selinux\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989858 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-system-cni-dir\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989873 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-run-ovn-kubernetes\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.991639 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989899 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0219d35-4a8d-4383-acda-cc5fad912ade-agent-certs\") pod \"konnectivity-agent-hjb7c\" (UID: \"b0219d35-4a8d-4383-acda-cc5fad912ade\") " pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989528 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be0f4959-65ba-40a2-aca1-76b571c2ce30-cni-binary-copy\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989955 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brhqt\" (UniqueName: \"kubernetes.io/projected/08095808-76be-4f3c-8ff2-c7dfaeb5c832-kube-api-access-brhqt\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989981 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-var-lib-kubelet\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.989977 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/25861972-bb71-4dc8-bfae-31d8c5a5bd8e-serviceca\") pod \"node-ca-zmxqf\" (UID: \"25861972-bb71-4dc8-bfae-31d8c5a5bd8e\") " pod="openshift-image-registry/node-ca-zmxqf" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990007 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/822bda63-ed87-40ff-86a1-5ebee377ed5a-hosts-file\") pod \"node-resolver-bkxdh\" (UID: \"822bda63-ed87-40ff-86a1-5ebee377ed5a\") " pod="openshift-dns/node-resolver-bkxdh" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990032 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-etc-openvswitch\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990048 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be0f4959-65ba-40a2-aca1-76b571c2ce30-host-var-lib-kubelet\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-cni-bin\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990094 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-cni-netd\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990116 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/822bda63-ed87-40ff-86a1-5ebee377ed5a-hosts-file\") pod \"node-resolver-bkxdh\" (UID: \"822bda63-ed87-40ff-86a1-5ebee377ed5a\") " pod="openshift-dns/node-resolver-bkxdh" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-socket-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990168 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvtn\" (UniqueName: \"kubernetes.io/projected/822bda63-ed87-40ff-86a1-5ebee377ed5a-kube-api-access-9vvtn\") pod \"node-resolver-bkxdh\" (UID: \"822bda63-ed87-40ff-86a1-5ebee377ed5a\") " pod="openshift-dns/node-resolver-bkxdh" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990190 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-log-socket\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990206 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqv4v\" (UniqueName: \"kubernetes.io/projected/74c2edb1-405d-4d3d-ae3f-d07b87d495f9-kube-api-access-nqv4v\") pod \"iptables-alerter-k2t8t\" (UID: \"74c2edb1-405d-4d3d-ae3f-d07b87d495f9\") " pod="openshift-network-operator/iptables-alerter-k2t8t" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990212 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08095808-76be-4f3c-8ff2-c7dfaeb5c832-socket-dir\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990245 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-sysctl-conf\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.992276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-systemd\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.992882 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990289 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-run\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.992882 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990334 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-sysctl-conf\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.992882 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990363 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-run\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.992882 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.990377 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/722e256c-433a-4b5f-aba9-960dd03efc28-etc-systemd\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.992882 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.992789 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/722e256c-433a-4b5f-aba9-960dd03efc28-tmp\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.992882 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.992864 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/722e256c-433a-4b5f-aba9-960dd03efc28-etc-tuned\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:12.997492 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.997471 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:20:12.997589 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.997496 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:20:12.997589 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.997508 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mnbn7 for pod openshift-network-diagnostics/network-check-target-m6fn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:12.997589 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:12.997554 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7 podName:f8064e28-3a81-423c-b1b4-3610c44f0f46 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:13.497537808 +0000 UTC m=+2.981936668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mnbn7" (UniqueName: "kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7") pod "network-check-target-m6fn6" (UID: "f8064e28-3a81-423c-b1b4-3610c44f0f46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:12.999807 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:12.999778 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqzzp\" (UniqueName: \"kubernetes.io/projected/80d9b28a-6045-49bf-8f48-304a8396799f-kube-api-access-hqzzp\") pod \"multus-additional-cni-plugins-6qdwq\" (UID: \"80d9b28a-6045-49bf-8f48-304a8396799f\") " pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:13.000560 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.000528 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kklm5\" (UniqueName: \"kubernetes.io/projected/be0f4959-65ba-40a2-aca1-76b571c2ce30-kube-api-access-kklm5\") pod \"multus-hg6x5\" (UID: \"be0f4959-65ba-40a2-aca1-76b571c2ce30\") " pod="openshift-multus/multus-hg6x5" Jun 08 11:20:13.000765 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.000737 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79hjp\" (UniqueName: \"kubernetes.io/projected/25861972-bb71-4dc8-bfae-31d8c5a5bd8e-kube-api-access-79hjp\") pod \"node-ca-zmxqf\" (UID: \"25861972-bb71-4dc8-bfae-31d8c5a5bd8e\") " pod="openshift-image-registry/node-ca-zmxqf" Jun 08 11:20:13.001001 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.000984 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68kmp\" (UniqueName: \"kubernetes.io/projected/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-kube-api-access-68kmp\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:13.001384 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.001352 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brhqt\" (UniqueName: \"kubernetes.io/projected/08095808-76be-4f3c-8ff2-c7dfaeb5c832-kube-api-access-brhqt\") pod \"aws-ebs-csi-driver-node-htqfd\" (UID: \"08095808-76be-4f3c-8ff2-c7dfaeb5c832\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:13.002735 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.002714 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvtn\" (UniqueName: \"kubernetes.io/projected/822bda63-ed87-40ff-86a1-5ebee377ed5a-kube-api-access-9vvtn\") pod \"node-resolver-bkxdh\" (UID: \"822bda63-ed87-40ff-86a1-5ebee377ed5a\") " pod="openshift-dns/node-resolver-bkxdh" Jun 08 11:20:13.002803 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.002734 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqn5l\" (UniqueName: \"kubernetes.io/projected/722e256c-433a-4b5f-aba9-960dd03efc28-kube-api-access-xqn5l\") pod \"tuned-2m76c\" (UID: \"722e256c-433a-4b5f-aba9-960dd03efc28\") " pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:13.090720 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090679 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-systemd-units\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.090888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-run-netns\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.090888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090764 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-run-ovn\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.090888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090771 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-systemd-units\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.090888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090792 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-var-lib-openvswitch\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.090888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090848 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-var-lib-openvswitch\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.090888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090845 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-run-netns\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.090888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090872 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-kubelet\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-slash\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090907 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-kubelet\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090846 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-run-ovn\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090920 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0219d35-4a8d-4383-acda-cc5fad912ade-konnectivity-ca\") pod \"konnectivity-agent-hjb7c\" (UID: \"b0219d35-4a8d-4383-acda-cc5fad912ade\") " pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090971 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-slash\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.090971 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/74c2edb1-405d-4d3d-ae3f-d07b87d495f9-iptables-alerter-script\") pod \"iptables-alerter-k2t8t\" (UID: \"74c2edb1-405d-4d3d-ae3f-d07b87d495f9\") " pod="openshift-network-operator/iptables-alerter-k2t8t" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091022 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-dbus\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091046 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6534a1c3-5859-4fc5-8b05-ca877b16aa77-env-overrides\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091072 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74c2edb1-405d-4d3d-ae3f-d07b87d495f9-host-slash\") pod \"iptables-alerter-k2t8t\" (UID: \"74c2edb1-405d-4d3d-ae3f-d07b87d495f9\") " pod="openshift-network-operator/iptables-alerter-k2t8t" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091098 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6534a1c3-5859-4fc5-8b05-ca877b16aa77-ovn-node-metrics-cert\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091128 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-kubelet-config\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6ld\" (UniqueName: \"kubernetes.io/projected/6534a1c3-5859-4fc5-8b05-ca877b16aa77-kube-api-access-9m6ld\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091178 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091202 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-run-openvswitch\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-run-ovn-kubernetes\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091256 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-dbus\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:13.091280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0219d35-4a8d-4383-acda-cc5fad912ade-agent-certs\") pod \"konnectivity-agent-hjb7c\" (UID: \"b0219d35-4a8d-4383-acda-cc5fad912ade\") " pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091309 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-etc-openvswitch\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091339 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-cni-bin\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-cni-netd\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091383 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-log-socket\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqv4v\" (UniqueName: \"kubernetes.io/projected/74c2edb1-405d-4d3d-ae3f-d07b87d495f9-kube-api-access-nqv4v\") pod \"iptables-alerter-k2t8t\" (UID: \"74c2edb1-405d-4d3d-ae3f-d07b87d495f9\") " pod="openshift-network-operator/iptables-alerter-k2t8t" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091408 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-etc-openvswitch\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091430 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-node-log\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091459 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-cni-netd\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-run-systemd\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091484 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/74c2edb1-405d-4d3d-ae3f-d07b87d495f9-iptables-alerter-script\") pod \"iptables-alerter-k2t8t\" (UID: \"74c2edb1-405d-4d3d-ae3f-d07b87d495f9\") " pod="openshift-network-operator/iptables-alerter-k2t8t" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6534a1c3-5859-4fc5-8b05-ca877b16aa77-ovnkube-script-lib\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091500 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-cni-bin\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091509 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0219d35-4a8d-4383-acda-cc5fad912ade-konnectivity-ca\") pod \"konnectivity-agent-hjb7c\" (UID: \"b0219d35-4a8d-4383-acda-cc5fad912ade\") " pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091532 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6534a1c3-5859-4fc5-8b05-ca877b16aa77-env-overrides\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091537 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091559 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6534a1c3-5859-4fc5-8b05-ca877b16aa77-ovnkube-config\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092185 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091641 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091688 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-run-systemd\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092969 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:13.091716 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:13.092969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091727 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-run-openvswitch\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092969 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:13.091767 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret podName:579ebcf9-8227-48b0-ac5a-1d9e2b8504fb nodeName:}" failed. No retries permitted until 2026-06-08 11:20:13.591751053 +0000 UTC m=+3.076149910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret") pod "global-pull-secret-syncer-dvnth" (UID: "579ebcf9-8227-48b0-ac5a-1d9e2b8504fb") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:13.092969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091773 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-kubelet-config\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:13.092969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091814 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74c2edb1-405d-4d3d-ae3f-d07b87d495f9-host-slash\") pod \"iptables-alerter-k2t8t\" (UID: \"74c2edb1-405d-4d3d-ae3f-d07b87d495f9\") " pod="openshift-network-operator/iptables-alerter-k2t8t" Jun 08 11:20:13.092969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091816 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-host-run-ovn-kubernetes\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091850 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-log-socket\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091932 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6534a1c3-5859-4fc5-8b05-ca877b16aa77-node-log\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.091944 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6534a1c3-5859-4fc5-8b05-ca877b16aa77-ovnkube-script-lib\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.092969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.092083 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6534a1c3-5859-4fc5-8b05-ca877b16aa77-ovnkube-config\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.093852 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.093834 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6534a1c3-5859-4fc5-8b05-ca877b16aa77-ovn-node-metrics-cert\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.094012 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.093992 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0219d35-4a8d-4383-acda-cc5fad912ade-agent-certs\") pod \"konnectivity-agent-hjb7c\" (UID: \"b0219d35-4a8d-4383-acda-cc5fad912ade\") " pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:13.101475 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.101453 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6ld\" (UniqueName: \"kubernetes.io/projected/6534a1c3-5859-4fc5-8b05-ca877b16aa77-kube-api-access-9m6ld\") pod \"ovnkube-node-bx9jt\" (UID: \"6534a1c3-5859-4fc5-8b05-ca877b16aa77\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.101591 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.101453 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqv4v\" (UniqueName: \"kubernetes.io/projected/74c2edb1-405d-4d3d-ae3f-d07b87d495f9-kube-api-access-nqv4v\") pod \"iptables-alerter-k2t8t\" (UID: \"74c2edb1-405d-4d3d-ae3f-d07b87d495f9\") " pod="openshift-network-operator/iptables-alerter-k2t8t" Jun 08 11:20:13.179654 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.179555 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hg6x5" Jun 08 11:20:13.186183 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.186154 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" Jun 08 11:20:13.194395 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.194372 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2m76c" Jun 08 11:20:13.199084 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.199064 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bkxdh" Jun 08 11:20:13.204574 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.204556 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zmxqf" Jun 08 11:20:13.211015 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.210997 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6qdwq" Jun 08 11:20:13.217740 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.217723 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:13.226209 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.226192 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k2t8t" Jun 08 11:20:13.229778 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.229752 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:13.494628 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.494551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:13.494760 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:13.494671 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:13.494760 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:13.494722 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs podName:2413fc4e-6ee1-47a8-8621-657e43ccfcf4 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:14.49470851 +0000 UTC m=+3.979107351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs") pod "network-metrics-daemon-kthbk" (UID: "2413fc4e-6ee1-47a8-8621-657e43ccfcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:13.595289 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.595259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbn7\" (UniqueName: \"kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7\") pod \"network-check-target-m6fn6\" (UID: \"f8064e28-3a81-423c-b1b4-3610c44f0f46\") " pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:13.595446 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.595325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:13.595446 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:13.595427 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:13.595560 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:13.595457 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:20:13.595560 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:13.595473 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:20:13.595560 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:13.595486 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mnbn7 for pod openshift-network-diagnostics/network-check-target-m6fn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:13.595560 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:13.595490 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret podName:579ebcf9-8227-48b0-ac5a-1d9e2b8504fb nodeName:}" failed. No retries permitted until 2026-06-08 11:20:14.595471765 +0000 UTC m=+4.079870619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret") pod "global-pull-secret-syncer-dvnth" (UID: "579ebcf9-8227-48b0-ac5a-1d9e2b8504fb") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:13.595560 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:13.595521 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7 podName:f8064e28-3a81-423c-b1b4-3610c44f0f46 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:14.595510252 +0000 UTC m=+4.079909106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnbn7" (UniqueName: "kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7") pod "network-check-target-m6fn6" (UID: "f8064e28-3a81-423c-b1b4-3610c44f0f46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:13.665731 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:13.665687 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822bda63_ed87_40ff_86a1_5ebee377ed5a.slice/crio-0ca2089d88cb7e2d4571d668d6598f2b3b7aa0870715a1b0ee7f32b644a098dd WatchSource:0}: Error finding container 0ca2089d88cb7e2d4571d668d6598f2b3b7aa0870715a1b0ee7f32b644a098dd: Status 404 returned error can't find the container with id 0ca2089d88cb7e2d4571d668d6598f2b3b7aa0870715a1b0ee7f32b644a098dd Jun 08 11:20:13.666835 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:13.666815 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6534a1c3_5859_4fc5_8b05_ca877b16aa77.slice/crio-051ffb8b8abd32851188fde4250ac31dfcaa4c206ae5ad0ddaad1e22dd18aec9 WatchSource:0}: Error finding container 051ffb8b8abd32851188fde4250ac31dfcaa4c206ae5ad0ddaad1e22dd18aec9: Status 404 returned error can't find the container with id 051ffb8b8abd32851188fde4250ac31dfcaa4c206ae5ad0ddaad1e22dd18aec9 Jun 08 11:20:13.668198 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:13.668178 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722e256c_433a_4b5f_aba9_960dd03efc28.slice/crio-d28d96399fc1c88592d74780ae26ae0ba16044f8cb07ad601d7ec325382bf634 WatchSource:0}: Error finding container d28d96399fc1c88592d74780ae26ae0ba16044f8cb07ad601d7ec325382bf634: Status 404 returned error can't find the container with id d28d96399fc1c88592d74780ae26ae0ba16044f8cb07ad601d7ec325382bf634 Jun 08 11:20:13.672305 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:13.672277 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c2edb1_405d_4d3d_ae3f_d07b87d495f9.slice/crio-19de13fc24d5c93a096625b9db43b79a56d2720cb4268b8c52334580876bcf4d WatchSource:0}: Error finding container 19de13fc24d5c93a096625b9db43b79a56d2720cb4268b8c52334580876bcf4d: Status 404 returned error can't find the container with id 19de13fc24d5c93a096625b9db43b79a56d2720cb4268b8c52334580876bcf4d Jun 08 11:20:13.674946 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:13.674923 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08095808_76be_4f3c_8ff2_c7dfaeb5c832.slice/crio-a4bd6e2bb90ac5f1ca7ce47e3cffbb537a9fedc872bf56868790ba14bac9feba WatchSource:0}: Error finding container a4bd6e2bb90ac5f1ca7ce47e3cffbb537a9fedc872bf56868790ba14bac9feba: Status 404 returned error can't find the container with id a4bd6e2bb90ac5f1ca7ce47e3cffbb537a9fedc872bf56868790ba14bac9feba Jun 08 11:20:13.675651 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:13.675598 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0f4959_65ba_40a2_aca1_76b571c2ce30.slice/crio-f6129c247c37a33f95f1f1171e2ed30628d1d76311f138206557d68b955822be WatchSource:0}: Error finding container f6129c247c37a33f95f1f1171e2ed30628d1d76311f138206557d68b955822be: Status 404 returned error can't find the container with id f6129c247c37a33f95f1f1171e2ed30628d1d76311f138206557d68b955822be Jun 08 11:20:13.676449 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:13.676421 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d9b28a_6045_49bf_8f48_304a8396799f.slice/crio-9cb49abc651e476ae9f58a5d57d4a1f77199f100b82f0acb89cdcc41ea889cd1 WatchSource:0}: Error finding container 9cb49abc651e476ae9f58a5d57d4a1f77199f100b82f0acb89cdcc41ea889cd1: Status 404 returned error can't find the container with id 9cb49abc651e476ae9f58a5d57d4a1f77199f100b82f0acb89cdcc41ea889cd1 Jun 08 11:20:13.677445 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:13.677274 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25861972_bb71_4dc8_bfae_31d8c5a5bd8e.slice/crio-740d877d97d1a7331bb5ae34175a501b77c08c166843e485ad1719241a5e7075 WatchSource:0}: Error finding container 740d877d97d1a7331bb5ae34175a501b77c08c166843e485ad1719241a5e7075: Status 404 returned error can't find the container with id 740d877d97d1a7331bb5ae34175a501b77c08c166843e485ad1719241a5e7075 Jun 08 11:20:13.919917 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.919725 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-07 11:15:12 +0000 UTC" deadline="2028-02-21 19:02:55.9767748 +0000 UTC" Jun 08 11:20:13.919917 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.919915 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14959h42m42.056863309s" Jun 08 11:20:13.955393 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.955365 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" event={"ID":"08095808-76be-4f3c-8ff2-c7dfaeb5c832","Type":"ContainerStarted","Data":"a4bd6e2bb90ac5f1ca7ce47e3cffbb537a9fedc872bf56868790ba14bac9feba"} Jun 08 11:20:13.956529 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.956503 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hjb7c" event={"ID":"b0219d35-4a8d-4383-acda-cc5fad912ade","Type":"ContainerStarted","Data":"eee5bd8c53599915694832ed346b8b84259bfada5da82d6f7df67fc36cbb3f90"} Jun 08 11:20:13.957729 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.957711 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2m76c" event={"ID":"722e256c-433a-4b5f-aba9-960dd03efc28","Type":"ContainerStarted","Data":"d28d96399fc1c88592d74780ae26ae0ba16044f8cb07ad601d7ec325382bf634"} Jun 08 11:20:13.958743 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.958721 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bkxdh" event={"ID":"822bda63-ed87-40ff-86a1-5ebee377ed5a","Type":"ContainerStarted","Data":"0ca2089d88cb7e2d4571d668d6598f2b3b7aa0870715a1b0ee7f32b644a098dd"} Jun 08 11:20:13.960434 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.960402 2569 generic.go:356] "Generic (PLEG): container finished" podID="d77966ae3dbd03711bb241eda3758fb8" containerID="cc80d103929065df82f012e8f1cd8218a5c36cd3aef2c48d1ef8dabfc57ae263" exitCode=0 Jun 08 11:20:13.960434 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.960431 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" event={"ID":"d77966ae3dbd03711bb241eda3758fb8","Type":"ContainerDied","Data":"cc80d103929065df82f012e8f1cd8218a5c36cd3aef2c48d1ef8dabfc57ae263"} Jun 08 11:20:13.961534 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.961516 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hg6x5" event={"ID":"be0f4959-65ba-40a2-aca1-76b571c2ce30","Type":"ContainerStarted","Data":"f6129c247c37a33f95f1f1171e2ed30628d1d76311f138206557d68b955822be"} Jun 08 11:20:13.962498 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.962479 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k2t8t" event={"ID":"74c2edb1-405d-4d3d-ae3f-d07b87d495f9","Type":"ContainerStarted","Data":"19de13fc24d5c93a096625b9db43b79a56d2720cb4268b8c52334580876bcf4d"} Jun 08 11:20:13.963465 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.963448 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" event={"ID":"6534a1c3-5859-4fc5-8b05-ca877b16aa77","Type":"ContainerStarted","Data":"051ffb8b8abd32851188fde4250ac31dfcaa4c206ae5ad0ddaad1e22dd18aec9"} Jun 08 11:20:13.964845 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.964815 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" event={"ID":"b69eb2fa1fe4d666208120158aec889b","Type":"ContainerStarted","Data":"2acecbe5a22a140f94249dc1865b9d84f684cd10ceec6e36294b7bc668346f7d"} Jun 08 11:20:13.965845 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.965827 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zmxqf" event={"ID":"25861972-bb71-4dc8-bfae-31d8c5a5bd8e","Type":"ContainerStarted","Data":"740d877d97d1a7331bb5ae34175a501b77c08c166843e485ad1719241a5e7075"} Jun 08 11:20:13.970043 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.970013 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qdwq" event={"ID":"80d9b28a-6045-49bf-8f48-304a8396799f","Type":"ContainerStarted","Data":"9cb49abc651e476ae9f58a5d57d4a1f77199f100b82f0acb89cdcc41ea889cd1"} Jun 08 11:20:13.989287 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:13.989241 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-94.ec2.internal" podStartSLOduration=1.9892281760000001 podStartE2EDuration="1.989228176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:13.988495245 +0000 UTC m=+3.472894130" watchObservedRunningTime="2026-06-08 11:20:13.989228176 +0000 UTC m=+3.473627037" Jun 08 11:20:14.511681 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:14.508346 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:14.511681 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:14.508502 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:14.511681 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:14.508565 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs podName:2413fc4e-6ee1-47a8-8621-657e43ccfcf4 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:16.508547371 +0000 UTC m=+5.992946215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs") pod "network-metrics-daemon-kthbk" (UID: "2413fc4e-6ee1-47a8-8621-657e43ccfcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:14.609328 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:14.609284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:14.609522 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:14.609349 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbn7\" (UniqueName: \"kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7\") pod \"network-check-target-m6fn6\" (UID: \"f8064e28-3a81-423c-b1b4-3610c44f0f46\") " pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:14.609584 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:14.609534 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:20:14.609584 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:14.609552 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:20:14.609584 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:14.609565 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mnbn7 for pod openshift-network-diagnostics/network-check-target-m6fn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:14.609758 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:14.609642 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7 podName:f8064e28-3a81-423c-b1b4-3610c44f0f46 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:16.609624291 +0000 UTC m=+6.094023148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnbn7" (UniqueName: "kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7") pod "network-check-target-m6fn6" (UID: "f8064e28-3a81-423c-b1b4-3610c44f0f46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:14.610049 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:14.610028 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:14.610123 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:14.610088 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret podName:579ebcf9-8227-48b0-ac5a-1d9e2b8504fb nodeName:}" failed. No retries permitted until 2026-06-08 11:20:16.610073169 +0000 UTC m=+6.094472012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret") pod "global-pull-secret-syncer-dvnth" (UID: "579ebcf9-8227-48b0-ac5a-1d9e2b8504fb") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:14.920647 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:14.920521 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-07 11:15:12 +0000 UTC" deadline="2028-03-17 17:09:21.187504092 +0000 UTC" Jun 08 11:20:14.920647 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:14.920583 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15557h49m6.266924898s" Jun 08 11:20:14.950320 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:14.950288 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:14.950320 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:14.950324 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:14.950593 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:14.950419 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:14.950593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:14.950296 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:14.950593 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:14.950521 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:14.950758 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:14.950594 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:14.989737 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:14.989668 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" event={"ID":"d77966ae3dbd03711bb241eda3758fb8","Type":"ContainerStarted","Data":"2228b72366d9a3d9c046bb2293a26813dbc35a911aba64871ba632987bcf554b"} Jun 08 11:20:16.526823 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:16.526787 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:16.527318 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:16.526929 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:16.527318 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:16.526989 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs podName:2413fc4e-6ee1-47a8-8621-657e43ccfcf4 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:20.526971005 +0000 UTC m=+10.011369862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs") pod "network-metrics-daemon-kthbk" (UID: "2413fc4e-6ee1-47a8-8621-657e43ccfcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:16.628864 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:16.628075 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:16.628864 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:16.628131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbn7\" (UniqueName: \"kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7\") pod \"network-check-target-m6fn6\" (UID: \"f8064e28-3a81-423c-b1b4-3610c44f0f46\") " pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:16.628864 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:16.628291 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:20:16.628864 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:16.628310 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:20:16.628864 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:16.628323 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mnbn7 for pod openshift-network-diagnostics/network-check-target-m6fn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:16.628864 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:16.628377 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7 podName:f8064e28-3a81-423c-b1b4-3610c44f0f46 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:20.628359924 +0000 UTC m=+10.112758768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnbn7" (UniqueName: "kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7") pod "network-check-target-m6fn6" (UID: "f8064e28-3a81-423c-b1b4-3610c44f0f46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:16.628864 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:16.628725 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:16.628864 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:16.628762 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret podName:579ebcf9-8227-48b0-ac5a-1d9e2b8504fb nodeName:}" failed. No retries permitted until 2026-06-08 11:20:20.628751362 +0000 UTC m=+10.113150203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret") pod "global-pull-secret-syncer-dvnth" (UID: "579ebcf9-8227-48b0-ac5a-1d9e2b8504fb") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:16.950243 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:16.950152 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:16.950395 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:16.950289 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:16.950902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:16.950653 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:16.950902 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:16.950743 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:16.950902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:16.950783 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:16.950902 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:16.950845 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:18.950955 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:18.950920 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:18.951394 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:18.951056 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:18.951459 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:18.951428 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:18.951576 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:18.951516 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:18.951665 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:18.951588 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:18.951719 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:18.951668 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:20.562731 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:20.562693 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:20.563183 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:20.562941 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:20.563183 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:20.563023 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs podName:2413fc4e-6ee1-47a8-8621-657e43ccfcf4 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:28.562986754 +0000 UTC m=+18.047385600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs") pod "network-metrics-daemon-kthbk" (UID: "2413fc4e-6ee1-47a8-8621-657e43ccfcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:20.663944 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:20.663893 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:20.664131 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:20.663957 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbn7\" (UniqueName: \"kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7\") pod \"network-check-target-m6fn6\" (UID: \"f8064e28-3a81-423c-b1b4-3610c44f0f46\") " pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:20.664131 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:20.664093 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:20:20.664131 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:20.664109 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:20:20.664131 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:20.664119 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mnbn7 for pod openshift-network-diagnostics/network-check-target-m6fn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:20.664335 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:20.664133 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:20.664335 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:20.664163 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7 podName:f8064e28-3a81-423c-b1b4-3610c44f0f46 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:28.664151281 +0000 UTC m=+18.148550134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnbn7" (UniqueName: "kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7") pod "network-check-target-m6fn6" (UID: "f8064e28-3a81-423c-b1b4-3610c44f0f46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:20.664335 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:20.664198 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret podName:579ebcf9-8227-48b0-ac5a-1d9e2b8504fb nodeName:}" failed. No retries permitted until 2026-06-08 11:20:28.664181089 +0000 UTC m=+18.148579932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret") pod "global-pull-secret-syncer-dvnth" (UID: "579ebcf9-8227-48b0-ac5a-1d9e2b8504fb") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:20.948883 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:20.948803 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:20.949026 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:20.948930 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:20.949310 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:20.949289 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:20.949430 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:20.949408 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:20.950398 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:20.950375 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:20.950511 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:20.950472 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:22.949184 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:22.948755 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:22.949184 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:22.948887 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:22.949184 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:22.948754 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:22.949184 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:22.949019 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:22.949184 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:22.949045 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:22.949184 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:22.949102 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:24.948284 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:24.947831 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:24.948284 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:24.947849 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:24.948284 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:24.947963 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:24.948284 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:24.948051 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:24.948284 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:24.948025 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:24.948284 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:24.948141 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:26.948388 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:26.948350 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:26.948842 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:26.948350 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:26.948842 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:26.948464 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:26.948842 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:26.948350 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:26.948842 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:26.948550 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:26.948842 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:26.948665 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:28.625365 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:28.625317 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:28.625885 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:28.625533 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:28.625885 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:28.625632 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs podName:2413fc4e-6ee1-47a8-8621-657e43ccfcf4 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:44.625594938 +0000 UTC m=+34.109993780 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs") pod "network-metrics-daemon-kthbk" (UID: "2413fc4e-6ee1-47a8-8621-657e43ccfcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:28.726475 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:28.726421 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:28.726475 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:28.726479 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbn7\" (UniqueName: \"kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7\") pod \"network-check-target-m6fn6\" (UID: \"f8064e28-3a81-423c-b1b4-3610c44f0f46\") " pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:28.726793 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:28.726641 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:20:28.726793 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:28.726659 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:20:28.726793 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:28.726668 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mnbn7 for pod openshift-network-diagnostics/network-check-target-m6fn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:28.726793 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:28.726720 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7 podName:f8064e28-3a81-423c-b1b4-3610c44f0f46 nodeName:}" failed. No retries permitted until 2026-06-08 11:20:44.726703526 +0000 UTC m=+34.211102386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnbn7" (UniqueName: "kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7") pod "network-check-target-m6fn6" (UID: "f8064e28-3a81-423c-b1b4-3610c44f0f46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:28.726935 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:28.726816 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:28.726935 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:28.726871 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret podName:579ebcf9-8227-48b0-ac5a-1d9e2b8504fb nodeName:}" failed. No retries permitted until 2026-06-08 11:20:44.72685774 +0000 UTC m=+34.211256580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret") pod "global-pull-secret-syncer-dvnth" (UID: "579ebcf9-8227-48b0-ac5a-1d9e2b8504fb") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:28.948679 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:28.948581 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:28.948818 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:28.948580 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:28.948818 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:28.948718 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:28.948818 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:28.948794 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:28.948916 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:28.948829 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:28.948916 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:28.948890 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:30.949211 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:30.949178 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:30.949702 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:30.949288 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:30.949702 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:30.949403 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:30.949702 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:30.949524 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:30.949702 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:30.949595 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:30.949918 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:30.949708 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:32.050759 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:32.050532 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" event={"ID":"08095808-76be-4f3c-8ff2-c7dfaeb5c832","Type":"ContainerStarted","Data":"dfccacf17318ee03c934d58dda1166f9829a5aa55111f2b8a713ecc6979649de"} Jun 08 11:20:32.052039 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:32.051999 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bkxdh" event={"ID":"822bda63-ed87-40ff-86a1-5ebee377ed5a","Type":"ContainerStarted","Data":"574cb879171ece8e91809442af0f0967f07c084c580a6a1e0b6184cebfe39883"} Jun 08 11:20:32.948847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:32.948655 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:32.949023 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:32.948656 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:32.949023 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:32.948947 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:32.949023 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:32.948674 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:32.949023 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:32.949003 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:32.949196 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:32.949087 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:33.055887 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.055861 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zmxqf" event={"ID":"25861972-bb71-4dc8-bfae-31d8c5a5bd8e","Type":"ContainerStarted","Data":"25a8983d00cacbaeab52509068483353dca30153a12cd6666fced44a8b9aa548"} Jun 08 11:20:33.057398 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.057376 2569 generic.go:356] "Generic (PLEG): container finished" podID="80d9b28a-6045-49bf-8f48-304a8396799f" containerID="6baf4dfdb1118b9970c68dca56ae167bea640f0634435d2c8ad58a7864d120ef" exitCode=0 Jun 08 11:20:33.057490 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.057440 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qdwq" event={"ID":"80d9b28a-6045-49bf-8f48-304a8396799f","Type":"ContainerDied","Data":"6baf4dfdb1118b9970c68dca56ae167bea640f0634435d2c8ad58a7864d120ef"} Jun 08 11:20:33.058858 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.058825 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hjb7c" event={"ID":"b0219d35-4a8d-4383-acda-cc5fad912ade","Type":"ContainerStarted","Data":"c0dd234a59d328a8a42ff35bcb9a7a1c20cba4a1b50ac30d86adbdaa8517d3d7"} Jun 08 11:20:33.060238 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.060214 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2m76c" event={"ID":"722e256c-433a-4b5f-aba9-960dd03efc28","Type":"ContainerStarted","Data":"6611c2e7192b7c301b2d7c46b51736833c4951719b0d9a759c3b03ec1099d8d7"} Jun 08 11:20:33.061501 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.061481 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hg6x5" event={"ID":"be0f4959-65ba-40a2-aca1-76b571c2ce30","Type":"ContainerStarted","Data":"00e618aff09fd7705bfb794420db440fdd5e511d6fe72c08798719725283bf77"} Jun 08 11:20:33.063960 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.063942 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" event={"ID":"6534a1c3-5859-4fc5-8b05-ca877b16aa77","Type":"ContainerStarted","Data":"42099d5c853b929267fd005ed74dfcecbdb6dd4a450fdf4a7e19ebdf542d47d3"} Jun 08 11:20:33.064034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.063992 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" event={"ID":"6534a1c3-5859-4fc5-8b05-ca877b16aa77","Type":"ContainerStarted","Data":"5d5b6fd284326485a96fadb65ee36785ad0b4c23a7608a259a5ca687bfaa978f"} Jun 08 11:20:33.064034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.064021 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" event={"ID":"6534a1c3-5859-4fc5-8b05-ca877b16aa77","Type":"ContainerStarted","Data":"44f6f5a17795d53545ae5be40c9f3ce4fbf79bb0d5df7866ebe90cb73ab67f00"} Jun 08 11:20:33.064105 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.064047 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" event={"ID":"6534a1c3-5859-4fc5-8b05-ca877b16aa77","Type":"ContainerStarted","Data":"cfff481039a6a3a3d96393cd588a593c8c7a1ac20679713048c15e37674b1bbc"} Jun 08 11:20:33.064105 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.064073 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" event={"ID":"6534a1c3-5859-4fc5-8b05-ca877b16aa77","Type":"ContainerStarted","Data":"56771a523510979b9098c373dab445d9db268dd1b6b5e4b3836eb4f735e5cd17"} Jun 08 11:20:33.070183 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.070146 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-94.ec2.internal" podStartSLOduration=21.070133727 podStartE2EDuration="21.070133727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:15.006158988 +0000 UTC m=+4.490557850" watchObservedRunningTime="2026-06-08 11:20:33.070133727 +0000 UTC m=+22.554532589" Jun 08 11:20:33.070377 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.070360 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zmxqf" podStartSLOduration=3.888588641 podStartE2EDuration="22.070353868s" totalImagesPullingTime="18.181765227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:33.069693588 +0000 UTC m=+22.554092449" watchObservedRunningTime="2026-06-08 11:20:33.070353868 +0000 UTC m=+22.554752741" Jun 08 11:20:33.083402 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.083358 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hjb7c" podStartSLOduration=3.9645934179999998 podStartE2EDuration="22.083297607s" totalImagesPullingTime="18.118704189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:33.083112398 +0000 UTC m=+22.567511260" watchObservedRunningTime="2026-06-08 11:20:33.083297607 +0000 UTC m=+22.567696538" Jun 08 11:20:33.100113 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.100069 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hg6x5" podStartSLOduration=3.903188897 podStartE2EDuration="22.100056544s" totalImagesPullingTime="18.196867647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:33.099747315 +0000 UTC m=+22.584146177" watchObservedRunningTime="2026-06-08 11:20:33.100056544 +0000 UTC m=+22.584455405" Jun 08 11:20:33.115691 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.115654 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2m76c" podStartSLOduration=3.9237261009999997 podStartE2EDuration="22.115641995s" totalImagesPullingTime="18.191915894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:33.115261319 +0000 UTC m=+22.599660182" watchObservedRunningTime="2026-06-08 11:20:33.115641995 +0000 UTC m=+22.600040856" Jun 08 11:20:33.150275 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.150214 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bkxdh" podStartSLOduration=3.9773905259999998 podStartE2EDuration="22.15019521s" totalImagesPullingTime="18.172804684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:33.129004333 +0000 UTC m=+22.613403194" watchObservedRunningTime="2026-06-08 11:20:33.15019521 +0000 UTC m=+22.634594073" Jun 08 11:20:33.505674 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.505599 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jun 08 11:20:33.953329 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.953192 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-08T11:20:33.505663005Z","UUID":"5c1a1826-e858-4f69-8f66-6f3508e7bb26","Handler":null,"Name":"","Endpoint":""} Jun 08 11:20:33.954927 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.954906 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jun 08 11:20:33.955053 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:33.954943 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jun 08 11:20:34.071823 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:34.071785 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" event={"ID":"08095808-76be-4f3c-8ff2-c7dfaeb5c832","Type":"ContainerStarted","Data":"4697b8b1dc655862fa7b6bf6792ae479cca02fd88649079d3206b3d53aecc2ce"} Jun 08 11:20:34.073664 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:34.073627 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k2t8t" event={"ID":"74c2edb1-405d-4d3d-ae3f-d07b87d495f9","Type":"ContainerStarted","Data":"2a70f531fe6935ffff12d2787ff6e6f7526edfe4773df81e2d5855f21eeeb327"} Jun 08 11:20:34.077477 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:34.077331 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" event={"ID":"6534a1c3-5859-4fc5-8b05-ca877b16aa77","Type":"ContainerStarted","Data":"204dd2b79d0482cc25ba546cae65d7924ac9c99c8ccc007fa1339b78bf253bf0"} Jun 08 11:20:34.089151 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:34.089105 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k2t8t" podStartSLOduration=4.895872722 podStartE2EDuration="23.089088302s" totalImagesPullingTime="18.19321558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:34.08851698 +0000 UTC m=+23.572915839" watchObservedRunningTime="2026-06-08 11:20:34.089088302 +0000 UTC m=+23.573487162" Jun 08 11:20:34.309046 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:34.309012 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:34.309658 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:34.309632 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:34.948307 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:34.948227 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:34.948307 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:34.948246 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:34.948508 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:34.948230 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:34.948508 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:34.948360 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:34.948508 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:34.948433 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:34.948677 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:34.948518 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:35.081047 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:35.081015 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" event={"ID":"08095808-76be-4f3c-8ff2-c7dfaeb5c832","Type":"ContainerStarted","Data":"83d30ff32258ce3f2b5f1cb7dfb3e98de7b7000f58d2d328f6da64dc57d9a8ab"} Jun 08 11:20:35.081470 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:35.081449 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:35.081666 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:35.081649 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hjb7c" Jun 08 11:20:35.097821 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:35.097758 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-htqfd" podStartSLOduration=3.810235252 podStartE2EDuration="24.097743464s" totalImagesPullingTime="20.287508212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:35.09697759 +0000 UTC m=+24.581376451" watchObservedRunningTime="2026-06-08 11:20:35.097743464 +0000 UTC m=+24.582142325" Jun 08 11:20:36.090543 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:36.090324 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" event={"ID":"6534a1c3-5859-4fc5-8b05-ca877b16aa77","Type":"ContainerStarted","Data":"6325d7e1c9bcc6f73d87e43ad6ea67d7a486c1128d192688f34763440409009d"} Jun 08 11:20:36.948148 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:36.948113 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:36.948357 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:36.948113 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:36.948357 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:36.948251 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:36.948357 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:36.948116 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:36.948357 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:36.948336 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:36.948549 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:36.948377 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:38.096960 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:38.096773 2569 generic.go:356] "Generic (PLEG): container finished" podID="80d9b28a-6045-49bf-8f48-304a8396799f" containerID="f8d778a025e7b4d53792ca63565881e4edd464bf7fc55a9ba9bf3764bb9b7b89" exitCode=0 Jun 08 11:20:38.097828 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:38.096848 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qdwq" event={"ID":"80d9b28a-6045-49bf-8f48-304a8396799f","Type":"ContainerDied","Data":"f8d778a025e7b4d53792ca63565881e4edd464bf7fc55a9ba9bf3764bb9b7b89"} Jun 08 11:20:38.100839 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:38.100814 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" event={"ID":"6534a1c3-5859-4fc5-8b05-ca877b16aa77","Type":"ContainerStarted","Data":"c3a59ed8bf109a042c80eb7aa7c030618a6c298612c649e7299c05906a47631f"} Jun 08 11:20:38.101146 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:38.101128 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:38.101249 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:38.101235 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:38.117431 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:38.117401 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:38.144122 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:38.144076 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" podStartSLOduration=8.538257702 podStartE2EDuration="27.144060774s" totalImagesPullingTime="18.605803072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:38.143183839 +0000 UTC m=+27.627582749" watchObservedRunningTime="2026-06-08 11:20:38.144060774 +0000 UTC m=+27.628459666" Jun 08 11:20:38.948298 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:38.948266 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:38.948463 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:38.948267 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:38.948463 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:38.948405 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:38.948463 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:38.948453 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:38.948632 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:38.948519 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:38.948632 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:38.948572 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:39.105913 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:39.105833 2569 generic.go:356] "Generic (PLEG): container finished" podID="80d9b28a-6045-49bf-8f48-304a8396799f" containerID="dec8e0c6e7215933aaec5cff4b4af9b712e34c11a305532803930a546d7762ee" exitCode=0 Jun 08 11:20:39.106312 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:39.105910 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qdwq" event={"ID":"80d9b28a-6045-49bf-8f48-304a8396799f","Type":"ContainerDied","Data":"dec8e0c6e7215933aaec5cff4b4af9b712e34c11a305532803930a546d7762ee"} Jun 08 11:20:39.106370 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:39.106329 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:39.122592 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:39.122570 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:20:39.458128 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:39.458045 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m6fn6"] Jun 08 11:20:39.458283 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:39.458161 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:39.458283 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:39.458265 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:39.461039 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:39.461014 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kthbk"] Jun 08 11:20:39.461182 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:39.461111 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:39.461241 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:39.461203 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:39.461801 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:39.461751 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dvnth"] Jun 08 11:20:39.461923 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:39.461835 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:39.461923 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:39.461912 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:40.110167 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:40.110082 2569 generic.go:356] "Generic (PLEG): container finished" podID="80d9b28a-6045-49bf-8f48-304a8396799f" containerID="76585b64b755ce5f9a94796ce21c2f90d271eabbc81fdbbba2e3261e74cc8daa" exitCode=0 Jun 08 11:20:40.110525 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:40.110170 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qdwq" event={"ID":"80d9b28a-6045-49bf-8f48-304a8396799f","Type":"ContainerDied","Data":"76585b64b755ce5f9a94796ce21c2f90d271eabbc81fdbbba2e3261e74cc8daa"} Jun 08 11:20:40.949667 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:40.949635 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:40.949858 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:40.949681 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:40.949858 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:40.949747 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:40.949858 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:40.949774 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:40.949858 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:40.949813 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:40.950000 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:40.949908 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:42.947987 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:42.947953 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:42.947987 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:42.947976 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:42.948678 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:42.948001 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:42.948678 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:42.948093 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:42.948678 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:42.948245 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:42.948678 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:42.948320 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:44.642223 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:44.642182 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:44.642749 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:44.642324 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:44.642749 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:44.642388 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs podName:2413fc4e-6ee1-47a8-8621-657e43ccfcf4 nodeName:}" failed. No retries permitted until 2026-06-08 11:21:16.642371852 +0000 UTC m=+66.126770692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs") pod "network-metrics-daemon-kthbk" (UID: "2413fc4e-6ee1-47a8-8621-657e43ccfcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:20:44.742704 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:44.742662 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:44.742880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:44.742717 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbn7\" (UniqueName: \"kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7\") pod \"network-check-target-m6fn6\" (UID: \"f8064e28-3a81-423c-b1b4-3610c44f0f46\") " pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:44.742880 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:44.742825 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:44.742880 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:44.742848 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:20:44.742880 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:44.742865 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:20:44.742880 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:44.742877 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mnbn7 for pod openshift-network-diagnostics/network-check-target-m6fn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:44.743142 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:44.742888 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret podName:579ebcf9-8227-48b0-ac5a-1d9e2b8504fb nodeName:}" failed. No retries permitted until 2026-06-08 11:21:16.742873025 +0000 UTC m=+66.227271870 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret") pod "global-pull-secret-syncer-dvnth" (UID: "579ebcf9-8227-48b0-ac5a-1d9e2b8504fb") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:20:44.743142 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:44.742926 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7 podName:f8064e28-3a81-423c-b1b4-3610c44f0f46 nodeName:}" failed. No retries permitted until 2026-06-08 11:21:16.742910823 +0000 UTC m=+66.227309681 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnbn7" (UniqueName: "kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7") pod "network-check-target-m6fn6" (UID: "f8064e28-3a81-423c-b1b4-3610c44f0f46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:20:44.948383 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:44.948303 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:44.948533 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:44.948427 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6fn6" podUID="f8064e28-3a81-423c-b1b4-3610c44f0f46" Jun 08 11:20:44.948533 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:44.948303 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:44.948663 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:44.948532 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:20:44.948663 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:44.948302 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:44.948663 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:44.948633 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvnth" podUID="579ebcf9-8227-48b0-ac5a-1d9e2b8504fb" Jun 08 11:20:45.815384 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.815315 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-94.ec2.internal" event="NodeReady" Jun 08 11:20:45.815833 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.815457 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jun 08 11:20:45.863117 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.863089 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jjjwt"] Jun 08 11:20:45.877902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.877877 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n5m8b"] Jun 08 11:20:45.878070 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.877982 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jjjwt" Jun 08 11:20:45.880844 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.880812 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jun 08 11:20:45.880844 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.880815 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6zq7l\"" Jun 08 11:20:45.881076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.880853 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jun 08 11:20:45.881076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.880871 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jun 08 11:20:45.899395 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.899372 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jjjwt"] Jun 08 11:20:45.899505 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.899402 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n5m8b"] Jun 08 11:20:45.899505 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.899482 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:45.901978 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.901961 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-flcnh\"" Jun 08 11:20:45.902085 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.902020 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jun 08 11:20:45.902085 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:45.902065 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jun 08 11:20:46.052328 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.052292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74ad6f9e-7b62-45d8-80b6-391b0f3758cb-cert\") pod \"ingress-canary-jjjwt\" (UID: \"74ad6f9e-7b62-45d8-80b6-391b0f3758cb\") " pod="openshift-ingress-canary/ingress-canary-jjjwt" Jun 08 11:20:46.052328 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.052327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.052516 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.052345 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-tmp-dir\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.052516 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.052388 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-config-volume\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.052516 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.052428 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr5xv\" (UniqueName: \"kubernetes.io/projected/74ad6f9e-7b62-45d8-80b6-391b0f3758cb-kube-api-access-rr5xv\") pod \"ingress-canary-jjjwt\" (UID: \"74ad6f9e-7b62-45d8-80b6-391b0f3758cb\") " pod="openshift-ingress-canary/ingress-canary-jjjwt" Jun 08 11:20:46.052516 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.052480 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgcd\" (UniqueName: \"kubernetes.io/projected/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-kube-api-access-7vgcd\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.129400 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.129297 2569 generic.go:356] "Generic (PLEG): container finished" podID="80d9b28a-6045-49bf-8f48-304a8396799f" containerID="81f13901a30bf3be4656f5eff9c4cc103b7c3ea55e96a50190753411e8f36e2b" exitCode=0 Jun 08 11:20:46.129400 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.129356 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qdwq" event={"ID":"80d9b28a-6045-49bf-8f48-304a8396799f","Type":"ContainerDied","Data":"81f13901a30bf3be4656f5eff9c4cc103b7c3ea55e96a50190753411e8f36e2b"} Jun 08 11:20:46.153010 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.152983 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr5xv\" (UniqueName: \"kubernetes.io/projected/74ad6f9e-7b62-45d8-80b6-391b0f3758cb-kube-api-access-rr5xv\") pod \"ingress-canary-jjjwt\" (UID: \"74ad6f9e-7b62-45d8-80b6-391b0f3758cb\") " pod="openshift-ingress-canary/ingress-canary-jjjwt" Jun 08 11:20:46.153145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.153029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgcd\" (UniqueName: \"kubernetes.io/projected/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-kube-api-access-7vgcd\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.153145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.153074 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74ad6f9e-7b62-45d8-80b6-391b0f3758cb-cert\") pod \"ingress-canary-jjjwt\" (UID: \"74ad6f9e-7b62-45d8-80b6-391b0f3758cb\") " pod="openshift-ingress-canary/ingress-canary-jjjwt" Jun 08 11:20:46.153145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.153089 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.153145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.153103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-tmp-dir\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.153145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.153119 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-config-volume\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.153354 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:46.153246 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:20:46.153354 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:46.153318 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls podName:483118b0-02a0-4b58-baf7-2e16a4fb6e2e nodeName:}" failed. No retries permitted until 2026-06-08 11:20:46.65329709 +0000 UTC m=+36.137695930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls") pod "dns-default-n5m8b" (UID: "483118b0-02a0-4b58-baf7-2e16a4fb6e2e") : secret "dns-default-metrics-tls" not found Jun 08 11:20:46.153569 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.153543 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-tmp-dir\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.153967 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.153948 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-config-volume\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.157622 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.157583 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74ad6f9e-7b62-45d8-80b6-391b0f3758cb-cert\") pod \"ingress-canary-jjjwt\" (UID: \"74ad6f9e-7b62-45d8-80b6-391b0f3758cb\") " pod="openshift-ingress-canary/ingress-canary-jjjwt" Jun 08 11:20:46.161593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.161552 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgcd\" (UniqueName: \"kubernetes.io/projected/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-kube-api-access-7vgcd\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.163820 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.163798 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr5xv\" (UniqueName: \"kubernetes.io/projected/74ad6f9e-7b62-45d8-80b6-391b0f3758cb-kube-api-access-rr5xv\") pod \"ingress-canary-jjjwt\" (UID: \"74ad6f9e-7b62-45d8-80b6-391b0f3758cb\") " pod="openshift-ingress-canary/ingress-canary-jjjwt" Jun 08 11:20:46.186731 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.186698 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jjjwt" Jun 08 11:20:46.381277 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.381058 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jjjwt"] Jun 08 11:20:46.384887 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:20:46.384862 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ad6f9e_7b62_45d8_80b6_391b0f3758cb.slice/crio-8a8a0ffa02af69e158c541dc7c468877311d356fcf8c7255d8e961d5cabe1ce3 WatchSource:0}: Error finding container 8a8a0ffa02af69e158c541dc7c468877311d356fcf8c7255d8e961d5cabe1ce3: Status 404 returned error can't find the container with id 8a8a0ffa02af69e158c541dc7c468877311d356fcf8c7255d8e961d5cabe1ce3 Jun 08 11:20:46.657903 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.657810 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:46.658070 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:46.657981 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:20:46.658070 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:46.658061 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls podName:483118b0-02a0-4b58-baf7-2e16a4fb6e2e nodeName:}" failed. No retries permitted until 2026-06-08 11:20:47.65803955 +0000 UTC m=+37.142438405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls") pod "dns-default-n5m8b" (UID: "483118b0-02a0-4b58-baf7-2e16a4fb6e2e") : secret "dns-default-metrics-tls" not found Jun 08 11:20:46.948825 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.948729 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:20:46.948825 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.948788 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:20:46.948825 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.948798 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:20:46.951902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.951882 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 08 11:20:46.951902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.951896 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 08 11:20:46.952105 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.951899 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l8wv8\"" Jun 08 11:20:46.952105 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.951923 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6vkk7\"" Jun 08 11:20:46.952225 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.952207 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 08 11:20:46.952662 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:46.952642 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 08 11:20:47.132775 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:47.132734 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jjjwt" event={"ID":"74ad6f9e-7b62-45d8-80b6-391b0f3758cb","Type":"ContainerStarted","Data":"8a8a0ffa02af69e158c541dc7c468877311d356fcf8c7255d8e961d5cabe1ce3"} Jun 08 11:20:47.136362 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:47.136335 2569 generic.go:356] "Generic (PLEG): container finished" podID="80d9b28a-6045-49bf-8f48-304a8396799f" containerID="3dfe5acc12665a105bf7a2cd3209bb56cc632b7743caf64e2b28178677636fa9" exitCode=0 Jun 08 11:20:47.136489 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:47.136375 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qdwq" event={"ID":"80d9b28a-6045-49bf-8f48-304a8396799f","Type":"ContainerDied","Data":"3dfe5acc12665a105bf7a2cd3209bb56cc632b7743caf64e2b28178677636fa9"} Jun 08 11:20:47.666148 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:47.665908 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:47.666287 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:47.666054 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:20:47.666287 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:47.666271 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls podName:483118b0-02a0-4b58-baf7-2e16a4fb6e2e nodeName:}" failed. No retries permitted until 2026-06-08 11:20:49.666245634 +0000 UTC m=+39.150644477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls") pod "dns-default-n5m8b" (UID: "483118b0-02a0-4b58-baf7-2e16a4fb6e2e") : secret "dns-default-metrics-tls" not found Jun 08 11:20:48.142146 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:48.142115 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qdwq" event={"ID":"80d9b28a-6045-49bf-8f48-304a8396799f","Type":"ContainerStarted","Data":"edda6671b88e25ee55fe7162b168e286a565a96066c9245b1ae4bfc3cd8b6f9c"} Jun 08 11:20:48.168656 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:48.168573 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6qdwq" podStartSLOduration=8.103143021 podStartE2EDuration="37.168547841s" totalImagesPullingTime="29.06540482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:48.167536022 +0000 UTC m=+37.651934888" watchObservedRunningTime="2026-06-08 11:20:48.168547841 +0000 UTC m=+37.652946703" Jun 08 11:20:49.145952 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:49.145914 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jjjwt" event={"ID":"74ad6f9e-7b62-45d8-80b6-391b0f3758cb","Type":"ContainerStarted","Data":"c1ee92044a140142fa4c5ae3ae6f5ad39f20b25e29ff8fb57b26a3f340a527c3"} Jun 08 11:20:49.162208 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:49.162168 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jjjwt" podStartSLOduration=2.157044411 podStartE2EDuration="4.162153881s" totalImagesPullingTime="2.00510947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:20:49.161724699 +0000 UTC m=+38.646123561" watchObservedRunningTime="2026-06-08 11:20:49.162153881 +0000 UTC m=+38.646552743" Jun 08 11:20:49.683363 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:49.683328 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:49.683539 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:49.683450 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:20:49.683539 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:49.683498 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls podName:483118b0-02a0-4b58-baf7-2e16a4fb6e2e nodeName:}" failed. No retries permitted until 2026-06-08 11:20:53.683486427 +0000 UTC m=+43.167885267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls") pod "dns-default-n5m8b" (UID: "483118b0-02a0-4b58-baf7-2e16a4fb6e2e") : secret "dns-default-metrics-tls" not found Jun 08 11:20:53.712381 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:20:53.712343 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:20:53.712887 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:53.712471 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:20:53.712887 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:20:53.712521 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls podName:483118b0-02a0-4b58-baf7-2e16a4fb6e2e nodeName:}" failed. No retries permitted until 2026-06-08 11:21:01.71250898 +0000 UTC m=+51.196907820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls") pod "dns-default-n5m8b" (UID: "483118b0-02a0-4b58-baf7-2e16a4fb6e2e") : secret "dns-default-metrics-tls" not found Jun 08 11:21:01.767716 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:01.767676 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:21:01.768363 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:21:01.767836 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:21:01.768363 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:21:01.767925 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls podName:483118b0-02a0-4b58-baf7-2e16a4fb6e2e nodeName:}" failed. No retries permitted until 2026-06-08 11:21:17.767904405 +0000 UTC m=+67.252303249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls") pod "dns-default-n5m8b" (UID: "483118b0-02a0-4b58-baf7-2e16a4fb6e2e") : secret "dns-default-metrics-tls" not found Jun 08 11:21:11.122691 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:11.122660 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx9jt" Jun 08 11:21:16.670019 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.669975 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:21:16.672752 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.672735 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 08 11:21:16.680815 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:21:16.680792 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 08 11:21:16.680880 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:21:16.680868 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs podName:2413fc4e-6ee1-47a8-8621-657e43ccfcf4 nodeName:}" failed. No retries permitted until 2026-06-08 11:22:20.680852366 +0000 UTC m=+130.165251206 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs") pod "network-metrics-daemon-kthbk" (UID: "2413fc4e-6ee1-47a8-8621-657e43ccfcf4") : secret "metrics-daemon-secret" not found Jun 08 11:21:16.770587 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.770552 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbn7\" (UniqueName: \"kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7\") pod \"network-check-target-m6fn6\" (UID: \"f8064e28-3a81-423c-b1b4-3610c44f0f46\") " pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:21:16.770703 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.770686 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:21:16.773335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.773307 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 08 11:21:16.773387 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.773312 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 08 11:21:16.783526 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.783504 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 08 11:21:16.785355 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.785337 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/579ebcf9-8227-48b0-ac5a-1d9e2b8504fb-original-pull-secret\") pod \"global-pull-secret-syncer-dvnth\" (UID: \"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb\") " pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:21:16.793847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.793815 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbn7\" (UniqueName: \"kubernetes.io/projected/f8064e28-3a81-423c-b1b4-3610c44f0f46-kube-api-access-mnbn7\") pod \"network-check-target-m6fn6\" (UID: \"f8064e28-3a81-423c-b1b4-3610c44f0f46\") " pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:21:16.963708 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.963638 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l8wv8\"" Jun 08 11:21:16.971733 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.971714 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:21:16.972950 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:16.972932 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvnth" Jun 08 11:21:17.109571 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:17.109536 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dvnth"] Jun 08 11:21:17.113698 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:21:17.113671 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579ebcf9_8227_48b0_ac5a_1d9e2b8504fb.slice/crio-9ff858c151ef561ea348d28f993608127a4b32655922f9816add145efba634be WatchSource:0}: Error finding container 9ff858c151ef561ea348d28f993608127a4b32655922f9816add145efba634be: Status 404 returned error can't find the container with id 9ff858c151ef561ea348d28f993608127a4b32655922f9816add145efba634be Jun 08 11:21:17.124307 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:17.124286 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m6fn6"] Jun 08 11:21:17.127555 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:21:17.127530 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8064e28_3a81_423c_b1b4_3610c44f0f46.slice/crio-23bea38c06d43ea699e3ee2327cbba547d6b2eaf9a6e19a5044b1b56a181dc32 WatchSource:0}: Error finding container 23bea38c06d43ea699e3ee2327cbba547d6b2eaf9a6e19a5044b1b56a181dc32: Status 404 returned error can't find the container with id 23bea38c06d43ea699e3ee2327cbba547d6b2eaf9a6e19a5044b1b56a181dc32 Jun 08 11:21:17.208744 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:17.208712 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dvnth" event={"ID":"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb","Type":"ContainerStarted","Data":"9ff858c151ef561ea348d28f993608127a4b32655922f9816add145efba634be"} Jun 08 11:21:17.209783 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:17.209761 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m6fn6" event={"ID":"f8064e28-3a81-423c-b1b4-3610c44f0f46","Type":"ContainerStarted","Data":"23bea38c06d43ea699e3ee2327cbba547d6b2eaf9a6e19a5044b1b56a181dc32"} Jun 08 11:21:17.778775 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:17.778728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:21:17.779231 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:21:17.778915 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:21:17.779231 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:21:17.778994 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls podName:483118b0-02a0-4b58-baf7-2e16a4fb6e2e nodeName:}" failed. No retries permitted until 2026-06-08 11:21:49.778973531 +0000 UTC m=+99.263372378 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls") pod "dns-default-n5m8b" (UID: "483118b0-02a0-4b58-baf7-2e16a4fb6e2e") : secret "dns-default-metrics-tls" not found Jun 08 11:21:22.225291 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:22.225250 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dvnth" event={"ID":"579ebcf9-8227-48b0-ac5a-1d9e2b8504fb","Type":"ContainerStarted","Data":"db53bef1d1ec27b7d99e4429a9551680a52cabb7d45e86ee222ec0fe86390687"} Jun 08 11:21:22.226604 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:22.226582 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m6fn6" event={"ID":"f8064e28-3a81-423c-b1b4-3610c44f0f46","Type":"ContainerStarted","Data":"e66592eb9f64b5b59be0b9b10b1916ec92c3343134bea8d658507498fa9f0653"} Jun 08 11:21:22.226735 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:22.226719 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:21:22.240988 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:22.240935 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dvnth" podStartSLOduration=66.785538795 podStartE2EDuration="1m11.240922417s" totalImagesPullingTime="4.455383622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:21:22.240575062 +0000 UTC m=+71.724973924" watchObservedRunningTime="2026-06-08 11:21:22.240922417 +0000 UTC m=+71.725321277" Jun 08 11:21:22.256175 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:22.256137 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m6fn6" podStartSLOduration=66.818724277 podStartE2EDuration="1m11.256128523s" totalImagesPullingTime="4.437404246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:21:22.255725467 +0000 UTC m=+71.740124330" watchObservedRunningTime="2026-06-08 11:21:22.256128523 +0000 UTC m=+71.740527383" Jun 08 11:21:49.801453 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:49.801411 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:21:49.801942 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:21:49.801520 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:21:49.801942 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:21:49.801585 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls podName:483118b0-02a0-4b58-baf7-2e16a4fb6e2e nodeName:}" failed. No retries permitted until 2026-06-08 11:22:53.801570488 +0000 UTC m=+163.285969329 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls") pod "dns-default-n5m8b" (UID: "483118b0-02a0-4b58-baf7-2e16a4fb6e2e") : secret "dns-default-metrics-tls" not found Jun 08 11:21:53.231843 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:21:53.231809 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m6fn6" Jun 08 11:22:20.716387 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:22:20.716341 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:22:20.716930 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:22:20.716496 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 08 11:22:20.716930 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:22:20.716569 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs podName:2413fc4e-6ee1-47a8-8621-657e43ccfcf4 nodeName:}" failed. No retries permitted until 2026-06-08 11:24:22.716550745 +0000 UTC m=+252.200949587 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs") pod "network-metrics-daemon-kthbk" (UID: "2413fc4e-6ee1-47a8-8621-657e43ccfcf4") : secret "metrics-daemon-secret" not found Jun 08 11:22:48.906859 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:22:48.906817 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n5m8b" podUID="483118b0-02a0-4b58-baf7-2e16a4fb6e2e" Jun 08 11:22:49.967297 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:22:49.967253 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kthbk" podUID="2413fc4e-6ee1-47a8-8621-657e43ccfcf4" Jun 08 11:22:53.840885 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:22:53.840795 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:22:53.841343 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:22:53.840915 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:22:53.841343 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:22:53.840978 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls podName:483118b0-02a0-4b58-baf7-2e16a4fb6e2e nodeName:}" failed. No retries permitted until 2026-06-08 11:24:55.84096406 +0000 UTC m=+285.325362902 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls") pod "dns-default-n5m8b" (UID: "483118b0-02a0-4b58-baf7-2e16a4fb6e2e") : secret "dns-default-metrics-tls" not found Jun 08 11:23:00.949483 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:00.949440 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n5m8b" Jun 08 11:23:03.948186 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:03.948147 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:23:21.274397 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.274338 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw"] Jun 08 11:23:21.276654 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.276633 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:21.278400 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.278361 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-896549976-8l62d"] Jun 08 11:23:21.279731 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.279707 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jun 08 11:23:21.280192 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.280172 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78dd696465-lcrhz"] Jun 08 11:23:21.280309 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.280213 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9njs4\"" Jun 08 11:23:21.280309 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.280226 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jun 08 11:23:21.280309 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.280250 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.280841 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.280824 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jun 08 11:23:21.282153 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.282132 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78dd696465-lcrhz" Jun 08 11:23:21.283833 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.283811 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jun 08 11:23:21.284226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.284208 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jun 08 11:23:21.284334 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.284211 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jun 08 11:23:21.284334 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.284280 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dpg2j\"" Jun 08 11:23:21.284465 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.284452 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jun 08 11:23:21.284542 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.284506 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jun 08 11:23:21.285390 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.285015 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-bmc8m\"" Jun 08 11:23:21.285390 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.285311 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jun 08 11:23:21.291193 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.291168 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jun 08 11:23:21.292022 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.291999 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw"] Jun 08 11:23:21.295092 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.295070 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-896549976-8l62d"] Jun 08 11:23:21.305568 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.305543 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78dd696465-lcrhz"] Jun 08 11:23:21.319827 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.319796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gftr7\" (UniqueName: \"kubernetes.io/projected/8341efd3-6f21-47f9-9b2c-bad74ab7935d-kube-api-access-gftr7\") pod \"volume-data-source-validator-78dd696465-lcrhz\" (UID: \"8341efd3-6f21-47f9-9b2c-bad74ab7935d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78dd696465-lcrhz" Jun 08 11:23:21.319962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.319832 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dzqjw\" (UID: \"fce44b84-bbb6-4d3a-9679-02bfa2d73a35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:21.319962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.319858 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc8847f9-8e90-498b-9c06-237949ccad50-tmp\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.319962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.319881 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fc8847f9-8e90-498b-9c06-237949ccad50-snapshots\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.319962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.319925 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc8847f9-8e90-498b-9c06-237949ccad50-service-ca-bundle\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.320212 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.319977 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zkd\" (UniqueName: \"kubernetes.io/projected/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-kube-api-access-26zkd\") pod \"cluster-samples-operator-854f8b96b6-dzqjw\" (UID: \"fce44b84-bbb6-4d3a-9679-02bfa2d73a35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:21.320212 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.320004 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc8847f9-8e90-498b-9c06-237949ccad50-trusted-ca-bundle\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.320212 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.320034 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhhm4\" (UniqueName: \"kubernetes.io/projected/fc8847f9-8e90-498b-9c06-237949ccad50-kube-api-access-zhhm4\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.320212 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.320096 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc8847f9-8e90-498b-9c06-237949ccad50-serving-cert\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.368870 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.368834 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-dbd9cf844-47jkz"] Jun 08 11:23:21.370888 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.370871 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-dbd9cf844-47jkz" Jun 08 11:23:21.373577 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.373553 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fnd5r\"" Jun 08 11:23:21.378523 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.378501 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d"] Jun 08 11:23:21.380458 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.380436 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6f975c8c69-c85sm"] Jun 08 11:23:21.380557 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.380547 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:21.382371 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.382352 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-dbd9cf844-47jkz"] Jun 08 11:23:21.382475 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.382433 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.383392 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.383371 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8vw4j\"" Jun 08 11:23:21.383759 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.383744 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jun 08 11:23:21.383843 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.383758 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jun 08 11:23:21.388932 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.388907 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jun 08 11:23:21.389042 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.388951 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jun 08 11:23:21.389042 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.388956 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cvmzf\"" Jun 08 11:23:21.389042 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.388966 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jun 08 11:23:21.389042 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.389014 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jun 08 11:23:21.389299 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.388973 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jun 08 11:23:21.391126 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.391106 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jun 08 11:23:21.404074 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.404048 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d"] Jun 08 11:23:21.404926 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.404904 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f975c8c69-c85sm"] Jun 08 11:23:21.420951 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.420922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhhm4\" (UniqueName: \"kubernetes.io/projected/fc8847f9-8e90-498b-9c06-237949ccad50-kube-api-access-zhhm4\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.421091 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.420957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ef8c72d1-5ce7-44c0-a259-cf8b40983674-nginx-conf\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:21.421091 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.420980 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.421091 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.420999 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz9ck\" (UniqueName: \"kubernetes.io/projected/2bcc00d8-28dc-4f53-ba05-4b53c852e037-kube-api-access-wz9ck\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.421091 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421079 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:21.421293 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q68b\" (UniqueName: \"kubernetes.io/projected/5bd062bd-d5b9-411a-8a07-52e26d6e2eca-kube-api-access-8q68b\") pod \"network-check-source-dbd9cf844-47jkz\" (UID: \"5bd062bd-d5b9-411a-8a07-52e26d6e2eca\") " pod="openshift-network-diagnostics/network-check-source-dbd9cf844-47jkz" Jun 08 11:23:21.421293 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc8847f9-8e90-498b-9c06-237949ccad50-serving-cert\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.421293 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gftr7\" (UniqueName: \"kubernetes.io/projected/8341efd3-6f21-47f9-9b2c-bad74ab7935d-kube-api-access-gftr7\") pod \"volume-data-source-validator-78dd696465-lcrhz\" (UID: \"8341efd3-6f21-47f9-9b2c-bad74ab7935d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78dd696465-lcrhz" Jun 08 11:23:21.421293 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421216 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dzqjw\" (UID: \"fce44b84-bbb6-4d3a-9679-02bfa2d73a35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:21.421293 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc8847f9-8e90-498b-9c06-237949ccad50-tmp\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.421293 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421253 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fc8847f9-8e90-498b-9c06-237949ccad50-snapshots\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.421293 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421268 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc8847f9-8e90-498b-9c06-237949ccad50-service-ca-bundle\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.421293 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421289 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-default-certificate\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.421686 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zkd\" (UniqueName: \"kubernetes.io/projected/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-kube-api-access-26zkd\") pod \"cluster-samples-operator-854f8b96b6-dzqjw\" (UID: \"fce44b84-bbb6-4d3a-9679-02bfa2d73a35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:21.421686 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421353 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc8847f9-8e90-498b-9c06-237949ccad50-trusted-ca-bundle\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.421686 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421379 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.421686 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421398 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-stats-auth\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.421686 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:21.421423 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 08 11:23:21.421686 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:21.421506 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls podName:fce44b84-bbb6-4d3a-9679-02bfa2d73a35 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:21.921484828 +0000 UTC m=+191.405883689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-dzqjw" (UID: "fce44b84-bbb6-4d3a-9679-02bfa2d73a35") : secret "samples-operator-tls" not found Jun 08 11:23:21.421899 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421698 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc8847f9-8e90-498b-9c06-237949ccad50-tmp\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.421952 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.421934 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fc8847f9-8e90-498b-9c06-237949ccad50-snapshots\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.422033 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.422011 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc8847f9-8e90-498b-9c06-237949ccad50-service-ca-bundle\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.422397 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.422381 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc8847f9-8e90-498b-9c06-237949ccad50-trusted-ca-bundle\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.423839 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.423813 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc8847f9-8e90-498b-9c06-237949ccad50-serving-cert\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.430550 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.430521 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zkd\" (UniqueName: \"kubernetes.io/projected/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-kube-api-access-26zkd\") pod \"cluster-samples-operator-854f8b96b6-dzqjw\" (UID: \"fce44b84-bbb6-4d3a-9679-02bfa2d73a35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:21.430750 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.430730 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftr7\" (UniqueName: \"kubernetes.io/projected/8341efd3-6f21-47f9-9b2c-bad74ab7935d-kube-api-access-gftr7\") pod \"volume-data-source-validator-78dd696465-lcrhz\" (UID: \"8341efd3-6f21-47f9-9b2c-bad74ab7935d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78dd696465-lcrhz" Jun 08 11:23:21.431061 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.431045 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhhm4\" (UniqueName: \"kubernetes.io/projected/fc8847f9-8e90-498b-9c06-237949ccad50-kube-api-access-zhhm4\") pod \"insights-operator-896549976-8l62d\" (UID: \"fc8847f9-8e90-498b-9c06-237949ccad50\") " pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.522481 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.522438 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.522481 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.522480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-stats-auth\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.522760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.522499 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ef8c72d1-5ce7-44c0-a259-cf8b40983674-nginx-conf\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:21.522760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.522527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.522760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.522550 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz9ck\" (UniqueName: \"kubernetes.io/projected/2bcc00d8-28dc-4f53-ba05-4b53c852e037-kube-api-access-wz9ck\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.522760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.522578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:21.522760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.522601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q68b\" (UniqueName: \"kubernetes.io/projected/5bd062bd-d5b9-411a-8a07-52e26d6e2eca-kube-api-access-8q68b\") pod \"network-check-source-dbd9cf844-47jkz\" (UID: \"5bd062bd-d5b9-411a-8a07-52e26d6e2eca\") " pod="openshift-network-diagnostics/network-check-source-dbd9cf844-47jkz" Jun 08 11:23:21.522760 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:21.522640 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle podName:2bcc00d8-28dc-4f53-ba05-4b53c852e037 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:22.022601587 +0000 UTC m=+191.507000450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle") pod "router-default-6f975c8c69-c85sm" (UID: "2bcc00d8-28dc-4f53-ba05-4b53c852e037") : configmap references non-existent config key: service-ca.crt Jun 08 11:23:21.522760 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:21.522713 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 08 11:23:21.522760 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:21.522735 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 08 11:23:21.522760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.522754 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-default-certificate\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.523213 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:21.522776 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert podName:ef8c72d1-5ce7-44c0-a259-cf8b40983674 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:22.02275831 +0000 UTC m=+191.507157165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert") pod "networking-console-plugin-d5d5bb78b-cnm4d" (UID: "ef8c72d1-5ce7-44c0-a259-cf8b40983674") : secret "networking-console-plugin-cert" not found Jun 08 11:23:21.523213 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:21.522796 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs podName:2bcc00d8-28dc-4f53-ba05-4b53c852e037 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:22.022786012 +0000 UTC m=+191.507184859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs") pod "router-default-6f975c8c69-c85sm" (UID: "2bcc00d8-28dc-4f53-ba05-4b53c852e037") : secret "router-metrics-certs-default" not found Jun 08 11:23:21.523331 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.523314 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ef8c72d1-5ce7-44c0-a259-cf8b40983674-nginx-conf\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:21.525136 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.525095 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-default-certificate\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.525211 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.525183 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-stats-auth\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.531778 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.531753 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz9ck\" (UniqueName: \"kubernetes.io/projected/2bcc00d8-28dc-4f53-ba05-4b53c852e037-kube-api-access-wz9ck\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:21.532116 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.532099 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q68b\" (UniqueName: \"kubernetes.io/projected/5bd062bd-d5b9-411a-8a07-52e26d6e2eca-kube-api-access-8q68b\") pod \"network-check-source-dbd9cf844-47jkz\" (UID: \"5bd062bd-d5b9-411a-8a07-52e26d6e2eca\") " pod="openshift-network-diagnostics/network-check-source-dbd9cf844-47jkz" Jun 08 11:23:21.597993 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.597967 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-896549976-8l62d" Jun 08 11:23:21.603265 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.603245 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78dd696465-lcrhz" Jun 08 11:23:21.679389 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.679359 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-dbd9cf844-47jkz" Jun 08 11:23:21.735775 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.735293 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-896549976-8l62d"] Jun 08 11:23:21.740711 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:23:21.740681 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8847f9_8e90_498b_9c06_237949ccad50.slice/crio-7299f6ab3d9eb479a96241707a07c467b887b51f9b1337d57498df4c6e497151 WatchSource:0}: Error finding container 7299f6ab3d9eb479a96241707a07c467b887b51f9b1337d57498df4c6e497151: Status 404 returned error can't find the container with id 7299f6ab3d9eb479a96241707a07c467b887b51f9b1337d57498df4c6e497151 Jun 08 11:23:21.750391 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.750368 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78dd696465-lcrhz"] Jun 08 11:23:21.753711 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:23:21.753686 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8341efd3_6f21_47f9_9b2c_bad74ab7935d.slice/crio-a1c308174edd55809c6f1c36f23813d8de004cca2adb7146803b376d31793d0f WatchSource:0}: Error finding container a1c308174edd55809c6f1c36f23813d8de004cca2adb7146803b376d31793d0f: Status 404 returned error can't find the container with id a1c308174edd55809c6f1c36f23813d8de004cca2adb7146803b376d31793d0f Jun 08 11:23:21.814796 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.814716 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-dbd9cf844-47jkz"] Jun 08 11:23:21.817399 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:23:21.817369 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd062bd_d5b9_411a_8a07_52e26d6e2eca.slice/crio-3a0d8438b3c82017f8966a7ef48489af4dc0e72e2a0cf2fdc171bf28bf30432e WatchSource:0}: Error finding container 3a0d8438b3c82017f8966a7ef48489af4dc0e72e2a0cf2fdc171bf28bf30432e: Status 404 returned error can't find the container with id 3a0d8438b3c82017f8966a7ef48489af4dc0e72e2a0cf2fdc171bf28bf30432e Jun 08 11:23:21.926034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:21.926000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dzqjw\" (UID: \"fce44b84-bbb6-4d3a-9679-02bfa2d73a35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:21.926212 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:21.926124 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 08 11:23:21.926212 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:21.926183 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls podName:fce44b84-bbb6-4d3a-9679-02bfa2d73a35 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:22.926166688 +0000 UTC m=+192.410565533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-dzqjw" (UID: "fce44b84-bbb6-4d3a-9679-02bfa2d73a35") : secret "samples-operator-tls" not found Jun 08 11:23:22.027277 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:22.027238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:22.027467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:22.027288 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:22.027467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:22.027326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:22.027467 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:22.027444 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 08 11:23:22.027605 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:22.027484 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 08 11:23:22.027605 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:22.027446 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle podName:2bcc00d8-28dc-4f53-ba05-4b53c852e037 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:23.027430507 +0000 UTC m=+192.511829348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle") pod "router-default-6f975c8c69-c85sm" (UID: "2bcc00d8-28dc-4f53-ba05-4b53c852e037") : configmap references non-existent config key: service-ca.crt Jun 08 11:23:22.027605 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:22.027514 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs podName:2bcc00d8-28dc-4f53-ba05-4b53c852e037 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:23.027499921 +0000 UTC m=+192.511898761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs") pod "router-default-6f975c8c69-c85sm" (UID: "2bcc00d8-28dc-4f53-ba05-4b53c852e037") : secret "router-metrics-certs-default" not found Jun 08 11:23:22.027605 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:22.027530 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert podName:ef8c72d1-5ce7-44c0-a259-cf8b40983674 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:23.027521837 +0000 UTC m=+192.511920678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert") pod "networking-console-plugin-d5d5bb78b-cnm4d" (UID: "ef8c72d1-5ce7-44c0-a259-cf8b40983674") : secret "networking-console-plugin-cert" not found Jun 08 11:23:22.508126 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:22.508016 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78dd696465-lcrhz" event={"ID":"8341efd3-6f21-47f9-9b2c-bad74ab7935d","Type":"ContainerStarted","Data":"a1c308174edd55809c6f1c36f23813d8de004cca2adb7146803b376d31793d0f"} Jun 08 11:23:22.509861 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:22.509831 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-8l62d" event={"ID":"fc8847f9-8e90-498b-9c06-237949ccad50","Type":"ContainerStarted","Data":"7299f6ab3d9eb479a96241707a07c467b887b51f9b1337d57498df4c6e497151"} Jun 08 11:23:22.511833 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:22.511391 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-dbd9cf844-47jkz" event={"ID":"5bd062bd-d5b9-411a-8a07-52e26d6e2eca","Type":"ContainerStarted","Data":"a121c71410eeeac41510aa60e584cca04e084a2b25306cdf28e306392ea35de8"} Jun 08 11:23:22.511833 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:22.511426 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-dbd9cf844-47jkz" event={"ID":"5bd062bd-d5b9-411a-8a07-52e26d6e2eca","Type":"ContainerStarted","Data":"3a0d8438b3c82017f8966a7ef48489af4dc0e72e2a0cf2fdc171bf28bf30432e"} Jun 08 11:23:22.530285 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:22.530221 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-dbd9cf844-47jkz" podStartSLOduration=1.530201336 podStartE2EDuration="1.530201336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:23:22.528874932 +0000 UTC m=+192.013273795" watchObservedRunningTime="2026-06-08 11:23:22.530201336 +0000 UTC m=+192.014600201" Jun 08 11:23:22.934252 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:22.934218 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dzqjw\" (UID: \"fce44b84-bbb6-4d3a-9679-02bfa2d73a35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:22.934421 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:22.934401 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 08 11:23:22.934493 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:22.934483 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls podName:fce44b84-bbb6-4d3a-9679-02bfa2d73a35 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:24.934461985 +0000 UTC m=+194.418860829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-dzqjw" (UID: "fce44b84-bbb6-4d3a-9679-02bfa2d73a35") : secret "samples-operator-tls" not found Jun 08 11:23:23.035546 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:23.035446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:23.035737 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:23.035587 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:23.035737 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:23.035643 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:23.035737 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:23.035650 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 08 11:23:23.035737 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:23.035725 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert podName:ef8c72d1-5ce7-44c0-a259-cf8b40983674 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:25.035706092 +0000 UTC m=+194.520104941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert") pod "networking-console-plugin-d5d5bb78b-cnm4d" (UID: "ef8c72d1-5ce7-44c0-a259-cf8b40983674") : secret "networking-console-plugin-cert" not found Jun 08 11:23:23.035956 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:23.035745 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle podName:2bcc00d8-28dc-4f53-ba05-4b53c852e037 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:25.035735994 +0000 UTC m=+194.520134835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle") pod "router-default-6f975c8c69-c85sm" (UID: "2bcc00d8-28dc-4f53-ba05-4b53c852e037") : configmap references non-existent config key: service-ca.crt Jun 08 11:23:23.035956 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:23.035753 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 08 11:23:23.035956 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:23.035802 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs podName:2bcc00d8-28dc-4f53-ba05-4b53c852e037 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:25.035787706 +0000 UTC m=+194.520186552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs") pod "router-default-6f975c8c69-c85sm" (UID: "2bcc00d8-28dc-4f53-ba05-4b53c852e037") : secret "router-metrics-certs-default" not found Jun 08 11:23:23.515442 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:23.515411 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-8l62d" event={"ID":"fc8847f9-8e90-498b-9c06-237949ccad50","Type":"ContainerStarted","Data":"b8965e2d05173cd48acf44bc5d58ac6483fd274b0206698999aab542724eee01"} Jun 08 11:23:23.516977 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:23.516955 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78dd696465-lcrhz" event={"ID":"8341efd3-6f21-47f9-9b2c-bad74ab7935d","Type":"ContainerStarted","Data":"248f4dd20b411927c219971609095398bd2e8a4b1586e49fd4e302bdc5134359"} Jun 08 11:23:23.550142 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:23.550082 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-896549976-8l62d" podStartSLOduration=0.852411848 podStartE2EDuration="2.550057128s" totalImagesPullingTime="1.69764528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:23:23.533833887 +0000 UTC m=+193.018232750" watchObservedRunningTime="2026-06-08 11:23:23.550057128 +0000 UTC m=+193.034455991" Jun 08 11:23:23.550782 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:23.550748 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78dd696465-lcrhz" podStartSLOduration=1.530507183 podStartE2EDuration="2.550737029s" totalImagesPullingTime="1.020229846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:23:23.549436464 +0000 UTC m=+193.033835328" watchObservedRunningTime="2026-06-08 11:23:23.550737029 +0000 UTC m=+193.035135895" Jun 08 11:23:24.952031 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:24.951989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dzqjw\" (UID: \"fce44b84-bbb6-4d3a-9679-02bfa2d73a35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:24.953121 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:24.952284 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 08 11:23:24.953121 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:24.952743 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls podName:fce44b84-bbb6-4d3a-9679-02bfa2d73a35 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:28.95271962 +0000 UTC m=+198.437118461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-dzqjw" (UID: "fce44b84-bbb6-4d3a-9679-02bfa2d73a35") : secret "samples-operator-tls" not found Jun 08 11:23:25.054006 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:25.053968 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:25.054161 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:25.054015 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:25.054161 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:25.054116 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 08 11:23:25.054281 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:25.054169 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs podName:2bcc00d8-28dc-4f53-ba05-4b53c852e037 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:29.054154419 +0000 UTC m=+198.538553260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs") pod "router-default-6f975c8c69-c85sm" (UID: "2bcc00d8-28dc-4f53-ba05-4b53c852e037") : secret "router-metrics-certs-default" not found Jun 08 11:23:25.054281 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:25.054189 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle podName:2bcc00d8-28dc-4f53-ba05-4b53c852e037 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:29.05417624 +0000 UTC m=+198.538575088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle") pod "router-default-6f975c8c69-c85sm" (UID: "2bcc00d8-28dc-4f53-ba05-4b53c852e037") : configmap references non-existent config key: service-ca.crt Jun 08 11:23:25.054281 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:25.054208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:25.054459 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:25.054364 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 08 11:23:25.054459 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:25.054413 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert podName:ef8c72d1-5ce7-44c0-a259-cf8b40983674 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:29.054398309 +0000 UTC m=+198.538797155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert") pod "networking-console-plugin-d5d5bb78b-cnm4d" (UID: "ef8c72d1-5ce7-44c0-a259-cf8b40983674") : secret "networking-console-plugin-cert" not found Jun 08 11:23:27.164448 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:27.164420 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bkxdh_822bda63-ed87-40ff-86a1-5ebee377ed5a/dns-node-resolver/0.log" Jun 08 11:23:28.163228 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:28.163199 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zmxqf_25861972-bb71-4dc8-bfae-31d8c5a5bd8e/node-ca/0.log" Jun 08 11:23:28.964515 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:28.964489 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jjjwt_74ad6f9e-7b62-45d8-80b6-391b0f3758cb/serve-healthcheck-canary/0.log" Jun 08 11:23:28.985857 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:28.985825 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dzqjw\" (UID: \"fce44b84-bbb6-4d3a-9679-02bfa2d73a35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:28.986009 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:28.985964 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 08 11:23:28.986053 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:28.986019 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls podName:fce44b84-bbb6-4d3a-9679-02bfa2d73a35 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:36.986005832 +0000 UTC m=+206.470404673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-dzqjw" (UID: "fce44b84-bbb6-4d3a-9679-02bfa2d73a35") : secret "samples-operator-tls" not found Jun 08 11:23:29.086971 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:29.086919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:29.087074 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:29.087059 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:29.087111 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:29.087074 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 08 11:23:29.087111 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:29.087099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:29.087187 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:29.087137 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert podName:ef8c72d1-5ce7-44c0-a259-cf8b40983674 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:37.087117982 +0000 UTC m=+206.571516829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert") pod "networking-console-plugin-d5d5bb78b-cnm4d" (UID: "ef8c72d1-5ce7-44c0-a259-cf8b40983674") : secret "networking-console-plugin-cert" not found Jun 08 11:23:29.087187 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:29.087180 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 08 11:23:29.087272 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:29.087214 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs podName:2bcc00d8-28dc-4f53-ba05-4b53c852e037 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:37.087202591 +0000 UTC m=+206.571601438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs") pod "router-default-6f975c8c69-c85sm" (UID: "2bcc00d8-28dc-4f53-ba05-4b53c852e037") : secret "router-metrics-certs-default" not found Jun 08 11:23:29.087272 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:29.087231 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle podName:2bcc00d8-28dc-4f53-ba05-4b53c852e037 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:37.087220978 +0000 UTC m=+206.571619819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle") pod "router-default-6f975c8c69-c85sm" (UID: "2bcc00d8-28dc-4f53-ba05-4b53c852e037") : configmap references non-existent config key: service-ca.crt Jun 08 11:23:37.051161 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.051119 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dzqjw\" (UID: \"fce44b84-bbb6-4d3a-9679-02bfa2d73a35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:37.053830 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.053807 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fce44b84-bbb6-4d3a-9679-02bfa2d73a35-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dzqjw\" (UID: \"fce44b84-bbb6-4d3a-9679-02bfa2d73a35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:37.151971 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.151930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:37.151971 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.151974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:37.152167 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.152002 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:37.152167 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:37.152137 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jun 08 11:23:37.152286 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:23:37.152205 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert podName:ef8c72d1-5ce7-44c0-a259-cf8b40983674 nodeName:}" failed. No retries permitted until 2026-06-08 11:23:53.152188368 +0000 UTC m=+222.636587210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert") pod "networking-console-plugin-d5d5bb78b-cnm4d" (UID: "ef8c72d1-5ce7-44c0-a259-cf8b40983674") : secret "networking-console-plugin-cert" not found Jun 08 11:23:37.152593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.152564 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bcc00d8-28dc-4f53-ba05-4b53c852e037-service-ca-bundle\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:37.154404 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.154384 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bcc00d8-28dc-4f53-ba05-4b53c852e037-metrics-certs\") pod \"router-default-6f975c8c69-c85sm\" (UID: \"2bcc00d8-28dc-4f53-ba05-4b53c852e037\") " pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:37.189001 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.188960 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" Jun 08 11:23:37.299140 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.299111 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:37.312976 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.312947 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw"] Jun 08 11:23:37.432170 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.432142 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f975c8c69-c85sm"] Jun 08 11:23:37.434824 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:23:37.434802 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcc00d8_28dc_4f53_ba05_4b53c852e037.slice/crio-7f2fa514d863ebe3c3e467af750cc0a903c01ffa845442d9db5af15a720fd742 WatchSource:0}: Error finding container 7f2fa514d863ebe3c3e467af750cc0a903c01ffa845442d9db5af15a720fd742: Status 404 returned error can't find the container with id 7f2fa514d863ebe3c3e467af750cc0a903c01ffa845442d9db5af15a720fd742 Jun 08 11:23:37.560085 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.559995 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f975c8c69-c85sm" event={"ID":"2bcc00d8-28dc-4f53-ba05-4b53c852e037","Type":"ContainerStarted","Data":"39086f863f8a022f31163afc4ae4d851cb8fbd10b5d82aed0c835ec31d095585"} Jun 08 11:23:37.560085 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.560055 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f975c8c69-c85sm" event={"ID":"2bcc00d8-28dc-4f53-ba05-4b53c852e037","Type":"ContainerStarted","Data":"7f2fa514d863ebe3c3e467af750cc0a903c01ffa845442d9db5af15a720fd742"} Jun 08 11:23:37.561125 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.561102 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" event={"ID":"fce44b84-bbb6-4d3a-9679-02bfa2d73a35","Type":"ContainerStarted","Data":"146f0dd1fc0822bb0b234a97644bc9dd0181216a5d2f9d0134d5816669e54d49"} Jun 08 11:23:37.578939 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:37.578892 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6f975c8c69-c85sm" podStartSLOduration=16.578879539 podStartE2EDuration="16.578879539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:23:37.578075668 +0000 UTC m=+207.062474530" watchObservedRunningTime="2026-06-08 11:23:37.578879539 +0000 UTC m=+207.063278402" Jun 08 11:23:38.300198 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:38.300154 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:38.302936 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:38.302912 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:38.564549 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:38.564438 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:38.566100 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:38.566072 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6f975c8c69-c85sm" Jun 08 11:23:39.568671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:39.568637 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" event={"ID":"fce44b84-bbb6-4d3a-9679-02bfa2d73a35","Type":"ContainerStarted","Data":"37d3f6c30c5ba7d2f573afefa7da4caee8b62c9c36c023c937d40453c96e1a86"} Jun 08 11:23:39.569050 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:39.568691 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" event={"ID":"fce44b84-bbb6-4d3a-9679-02bfa2d73a35","Type":"ContainerStarted","Data":"b470aa61737070efe6aebef337e5ee54005ce3a63e92090559b8ff2b17084272"} Jun 08 11:23:39.586058 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:39.586010 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dzqjw" podStartSLOduration=17.058489869 podStartE2EDuration="18.585995309s" totalImagesPullingTime="1.52750544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:23:39.585593174 +0000 UTC m=+209.069992036" watchObservedRunningTime="2026-06-08 11:23:39.585995309 +0000 UTC m=+209.070394171" Jun 08 11:23:48.642046 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.642010 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qnqgr"] Jun 08 11:23:48.648027 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.647998 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.651480 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.651453 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jun 08 11:23:48.651636 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.651454 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jun 08 11:23:48.651636 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.651489 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9gmff\"" Jun 08 11:23:48.658093 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.658066 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qnqgr"] Jun 08 11:23:48.704642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.704597 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65949fcfbf-49m7t"] Jun 08 11:23:48.708287 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.708260 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.710682 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.710660 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jun 08 11:23:48.710855 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.710839 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jun 08 11:23:48.710935 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.710847 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6625z\"" Jun 08 11:23:48.710935 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.710888 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jun 08 11:23:48.716442 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.716420 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jun 08 11:23:48.721628 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.721584 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65949fcfbf-49m7t"] Jun 08 11:23:48.741839 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.741806 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df10cd2f-a986-4b4d-bca6-f2fb37457b18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.741981 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.741878 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df10cd2f-a986-4b4d-bca6-f2fb37457b18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.741981 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.741917 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kv7v\" (UniqueName: \"kubernetes.io/projected/df10cd2f-a986-4b4d-bca6-f2fb37457b18-kube-api-access-5kv7v\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.741981 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.741945 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df10cd2f-a986-4b4d-bca6-f2fb37457b18-data-volume\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.742115 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.742005 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df10cd2f-a986-4b4d-bca6-f2fb37457b18-crio-socket\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.842591 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842550 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/177e0d64-9385-4ba2-a01f-d825fa7fddbf-installation-pull-secrets\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.842807 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842598 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df10cd2f-a986-4b4d-bca6-f2fb37457b18-data-volume\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.842807 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df10cd2f-a986-4b4d-bca6-f2fb37457b18-crio-socket\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.842807 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/177e0d64-9385-4ba2-a01f-d825fa7fddbf-ca-trust-extracted\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.842807 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842726 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/177e0d64-9385-4ba2-a01f-d825fa7fddbf-image-registry-private-configuration\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.842807 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842773 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df10cd2f-a986-4b4d-bca6-f2fb37457b18-crio-socket\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.842807 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842798 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/177e0d64-9385-4ba2-a01f-d825fa7fddbf-trusted-ca\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.843023 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842836 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/177e0d64-9385-4ba2-a01f-d825fa7fddbf-registry-tls\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.843023 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842861 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df10cd2f-a986-4b4d-bca6-f2fb37457b18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.843023 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/177e0d64-9385-4ba2-a01f-d825fa7fddbf-bound-sa-token\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.843023 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842945 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df10cd2f-a986-4b4d-bca6-f2fb37457b18-data-volume\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.843023 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842948 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxkb2\" (UniqueName: \"kubernetes.io/projected/177e0d64-9385-4ba2-a01f-d825fa7fddbf-kube-api-access-bxkb2\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.843023 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.842996 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df10cd2f-a986-4b4d-bca6-f2fb37457b18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.843229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.843022 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/177e0d64-9385-4ba2-a01f-d825fa7fddbf-registry-certificates\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.843229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.843056 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kv7v\" (UniqueName: \"kubernetes.io/projected/df10cd2f-a986-4b4d-bca6-f2fb37457b18-kube-api-access-5kv7v\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.843428 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.843410 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df10cd2f-a986-4b4d-bca6-f2fb37457b18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.845269 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.845250 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df10cd2f-a986-4b4d-bca6-f2fb37457b18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.851453 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.851433 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kv7v\" (UniqueName: \"kubernetes.io/projected/df10cd2f-a986-4b4d-bca6-f2fb37457b18-kube-api-access-5kv7v\") pod \"insights-runtime-extractor-qnqgr\" (UID: \"df10cd2f-a986-4b4d-bca6-f2fb37457b18\") " pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:48.944136 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.944057 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/177e0d64-9385-4ba2-a01f-d825fa7fddbf-ca-trust-extracted\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.944136 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.944104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/177e0d64-9385-4ba2-a01f-d825fa7fddbf-image-registry-private-configuration\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.944136 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.944137 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/177e0d64-9385-4ba2-a01f-d825fa7fddbf-trusted-ca\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.944387 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.944164 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/177e0d64-9385-4ba2-a01f-d825fa7fddbf-registry-tls\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.944387 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.944203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/177e0d64-9385-4ba2-a01f-d825fa7fddbf-bound-sa-token\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.944387 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.944236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxkb2\" (UniqueName: \"kubernetes.io/projected/177e0d64-9385-4ba2-a01f-d825fa7fddbf-kube-api-access-bxkb2\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.944387 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.944269 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/177e0d64-9385-4ba2-a01f-d825fa7fddbf-registry-certificates\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.944387 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.944302 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/177e0d64-9385-4ba2-a01f-d825fa7fddbf-installation-pull-secrets\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.944646 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.944501 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/177e0d64-9385-4ba2-a01f-d825fa7fddbf-ca-trust-extracted\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.945209 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.945179 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/177e0d64-9385-4ba2-a01f-d825fa7fddbf-trusted-ca\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.945338 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.945185 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/177e0d64-9385-4ba2-a01f-d825fa7fddbf-registry-certificates\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.947085 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.947066 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/177e0d64-9385-4ba2-a01f-d825fa7fddbf-registry-tls\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.947175 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.947084 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/177e0d64-9385-4ba2-a01f-d825fa7fddbf-installation-pull-secrets\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.947243 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.947222 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/177e0d64-9385-4ba2-a01f-d825fa7fddbf-image-registry-private-configuration\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.954922 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.954899 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/177e0d64-9385-4ba2-a01f-d825fa7fddbf-bound-sa-token\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.955130 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.955110 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxkb2\" (UniqueName: \"kubernetes.io/projected/177e0d64-9385-4ba2-a01f-d825fa7fddbf-kube-api-access-bxkb2\") pod \"image-registry-65949fcfbf-49m7t\" (UID: \"177e0d64-9385-4ba2-a01f-d825fa7fddbf\") " pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:48.957767 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:48.957751 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qnqgr" Jun 08 11:23:49.018809 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:49.018772 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:49.091554 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:49.091530 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qnqgr"] Jun 08 11:23:49.094886 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:23:49.094857 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf10cd2f_a986_4b4d_bca6_f2fb37457b18.slice/crio-5c8e1b88c707712cfe4fe1e99da370be527973a4aeef3c7ff30390544b544e4e WatchSource:0}: Error finding container 5c8e1b88c707712cfe4fe1e99da370be527973a4aeef3c7ff30390544b544e4e: Status 404 returned error can't find the container with id 5c8e1b88c707712cfe4fe1e99da370be527973a4aeef3c7ff30390544b544e4e Jun 08 11:23:49.155964 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:49.155935 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65949fcfbf-49m7t"] Jun 08 11:23:49.158108 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:23:49.158085 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177e0d64_9385_4ba2_a01f_d825fa7fddbf.slice/crio-6be7c7c68c565316ff5947270d97c09f8916611e64569c1814472b3c111fc484 WatchSource:0}: Error finding container 6be7c7c68c565316ff5947270d97c09f8916611e64569c1814472b3c111fc484: Status 404 returned error can't find the container with id 6be7c7c68c565316ff5947270d97c09f8916611e64569c1814472b3c111fc484 Jun 08 11:23:49.601233 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:49.601183 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnqgr" event={"ID":"df10cd2f-a986-4b4d-bca6-f2fb37457b18","Type":"ContainerStarted","Data":"5ce6476da407bc42ac9c872afdea3e8d7d64d2d6d6255034fca620b069903a10"} Jun 08 11:23:49.601431 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:49.601266 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnqgr" event={"ID":"df10cd2f-a986-4b4d-bca6-f2fb37457b18","Type":"ContainerStarted","Data":"5c8e1b88c707712cfe4fe1e99da370be527973a4aeef3c7ff30390544b544e4e"} Jun 08 11:23:49.602879 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:49.602850 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" event={"ID":"177e0d64-9385-4ba2-a01f-d825fa7fddbf","Type":"ContainerStarted","Data":"b86b3a4280e7c19b27dee73e206e23ec7a8bcccb011a741f8c05a969d35e24cb"} Jun 08 11:23:49.603017 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:49.602898 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" event={"ID":"177e0d64-9385-4ba2-a01f-d825fa7fddbf","Type":"ContainerStarted","Data":"6be7c7c68c565316ff5947270d97c09f8916611e64569c1814472b3c111fc484"} Jun 08 11:23:49.603017 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:49.602939 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:23:49.623183 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:49.623140 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" podStartSLOduration=1.62312483 podStartE2EDuration="1.62312483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:23:49.622908525 +0000 UTC m=+219.107307388" watchObservedRunningTime="2026-06-08 11:23:49.62312483 +0000 UTC m=+219.107523693" Jun 08 11:23:50.612855 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:50.612820 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnqgr" event={"ID":"df10cd2f-a986-4b4d-bca6-f2fb37457b18","Type":"ContainerStarted","Data":"7c02c42c85cdbfac4751663600bf6ac1209795dbd4223672b65ebf5130dcf419"} Jun 08 11:23:51.112116 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.112072 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6657875f97-g6lcj"] Jun 08 11:23:51.118934 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.118908 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6657875f97-g6lcj" Jun 08 11:23:51.121435 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.121410 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ll958\"" Jun 08 11:23:51.121435 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.121430 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jun 08 11:23:51.121650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.121459 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jun 08 11:23:51.124544 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.124473 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6657875f97-g6lcj"] Jun 08 11:23:51.263807 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.263772 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrf5p\" (UniqueName: \"kubernetes.io/projected/66448ff5-45a3-4749-b993-e446af7d3dd6-kube-api-access-zrf5p\") pod \"downloads-6657875f97-g6lcj\" (UID: \"66448ff5-45a3-4749-b993-e446af7d3dd6\") " pod="openshift-console/downloads-6657875f97-g6lcj" Jun 08 11:23:51.364282 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.364212 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrf5p\" (UniqueName: \"kubernetes.io/projected/66448ff5-45a3-4749-b993-e446af7d3dd6-kube-api-access-zrf5p\") pod \"downloads-6657875f97-g6lcj\" (UID: \"66448ff5-45a3-4749-b993-e446af7d3dd6\") " pod="openshift-console/downloads-6657875f97-g6lcj" Jun 08 11:23:51.372778 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.372754 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrf5p\" (UniqueName: \"kubernetes.io/projected/66448ff5-45a3-4749-b993-e446af7d3dd6-kube-api-access-zrf5p\") pod \"downloads-6657875f97-g6lcj\" (UID: \"66448ff5-45a3-4749-b993-e446af7d3dd6\") " pod="openshift-console/downloads-6657875f97-g6lcj" Jun 08 11:23:51.431287 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.431237 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6657875f97-g6lcj" Jun 08 11:23:51.561695 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.561657 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6657875f97-g6lcj"] Jun 08 11:23:51.565858 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:23:51.565814 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66448ff5_45a3_4749_b993_e446af7d3dd6.slice/crio-1cff58d34662ec85bb8d778f0fa366cd0b9feaa08d27f721838bff785af26301 WatchSource:0}: Error finding container 1cff58d34662ec85bb8d778f0fa366cd0b9feaa08d27f721838bff785af26301: Status 404 returned error can't find the container with id 1cff58d34662ec85bb8d778f0fa366cd0b9feaa08d27f721838bff785af26301 Jun 08 11:23:51.617101 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.617011 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6657875f97-g6lcj" event={"ID":"66448ff5-45a3-4749-b993-e446af7d3dd6","Type":"ContainerStarted","Data":"1cff58d34662ec85bb8d778f0fa366cd0b9feaa08d27f721838bff785af26301"} Jun 08 11:23:51.618879 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.618856 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnqgr" event={"ID":"df10cd2f-a986-4b4d-bca6-f2fb37457b18","Type":"ContainerStarted","Data":"59b9b4008bc808b88042891dcd9eb68422d03a8cdbea293b28bd8cf8a22269ba"} Jun 08 11:23:51.639097 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:51.639046 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qnqgr" podStartSLOduration=1.586087079 podStartE2EDuration="3.639031243s" totalImagesPullingTime="2.052944164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:23:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:23:51.637811598 +0000 UTC m=+221.122210459" watchObservedRunningTime="2026-06-08 11:23:51.639031243 +0000 UTC m=+221.123430105" Jun 08 11:23:53.180080 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:53.180035 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:53.182519 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:53.182497 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8c72d1-5ce7-44c0-a259-cf8b40983674-networking-console-plugin-cert\") pod \"networking-console-plugin-d5d5bb78b-cnm4d\" (UID: \"ef8c72d1-5ce7-44c0-a259-cf8b40983674\") " pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:53.189869 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:53.189847 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" Jun 08 11:23:53.342430 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:53.342394 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d"] Jun 08 11:23:53.345972 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:23:53.345944 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8c72d1_5ce7_44c0_a259_cf8b40983674.slice/crio-662a02e391d4d3be28d245305f72420be9b56b1edff316a94ea3b330f31e44b4 WatchSource:0}: Error finding container 662a02e391d4d3be28d245305f72420be9b56b1edff316a94ea3b330f31e44b4: Status 404 returned error can't find the container with id 662a02e391d4d3be28d245305f72420be9b56b1edff316a94ea3b330f31e44b4 Jun 08 11:23:53.627549 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:53.627514 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" event={"ID":"ef8c72d1-5ce7-44c0-a259-cf8b40983674","Type":"ContainerStarted","Data":"662a02e391d4d3be28d245305f72420be9b56b1edff316a94ea3b330f31e44b4"} Jun 08 11:23:55.587076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:55.587041 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7"] Jun 08 11:23:55.590354 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:55.590337 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7" Jun 08 11:23:55.592826 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:55.592806 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-g8jrx\"" Jun 08 11:23:55.592998 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:55.592983 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jun 08 11:23:55.600227 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:55.600205 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7"] Jun 08 11:23:55.634992 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:55.634955 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" event={"ID":"ef8c72d1-5ce7-44c0-a259-cf8b40983674","Type":"ContainerStarted","Data":"94c6c282c7d00b1a5f69950293d2e974563d65e5c9ddb765c97abf84ef5329c4"} Jun 08 11:23:55.652218 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:55.652170 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d5d5bb78b-cnm4d" podStartSLOduration=33.406888369 podStartE2EDuration="34.652156206s" totalImagesPullingTime="1.245267837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:23:55.651717683 +0000 UTC m=+225.136116549" watchObservedRunningTime="2026-06-08 11:23:55.652156206 +0000 UTC m=+225.136555069" Jun 08 11:23:55.699045 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:55.699007 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c74c35a7-f89f-45b9-b8c0-ca6f2f91c562-tls-certificates\") pod \"prometheus-operator-admission-webhook-5b6b8f594-zsdh7\" (UID: \"c74c35a7-f89f-45b9-b8c0-ca6f2f91c562\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7" Jun 08 11:23:55.800412 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:55.800376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c74c35a7-f89f-45b9-b8c0-ca6f2f91c562-tls-certificates\") pod \"prometheus-operator-admission-webhook-5b6b8f594-zsdh7\" (UID: \"c74c35a7-f89f-45b9-b8c0-ca6f2f91c562\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7" Jun 08 11:23:55.803404 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:55.803375 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c74c35a7-f89f-45b9-b8c0-ca6f2f91c562-tls-certificates\") pod \"prometheus-operator-admission-webhook-5b6b8f594-zsdh7\" (UID: \"c74c35a7-f89f-45b9-b8c0-ca6f2f91c562\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7" Jun 08 11:23:55.899698 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:55.899576 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7" Jun 08 11:23:56.046539 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:56.046498 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7"] Jun 08 11:23:56.062681 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:23:56.062646 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74c35a7_f89f_45b9_b8c0_ca6f2f91c562.slice/crio-6594b116a3e5321c4744b554b08e41e41874ce778a27c269cec8f3eca1705019 WatchSource:0}: Error finding container 6594b116a3e5321c4744b554b08e41e41874ce778a27c269cec8f3eca1705019: Status 404 returned error can't find the container with id 6594b116a3e5321c4744b554b08e41e41874ce778a27c269cec8f3eca1705019 Jun 08 11:23:56.641266 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:56.641211 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7" event={"ID":"c74c35a7-f89f-45b9-b8c0-ca6f2f91c562","Type":"ContainerStarted","Data":"6594b116a3e5321c4744b554b08e41e41874ce778a27c269cec8f3eca1705019"} Jun 08 11:23:57.645657 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:57.645603 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7" event={"ID":"c74c35a7-f89f-45b9-b8c0-ca6f2f91c562","Type":"ContainerStarted","Data":"b070cda208aa8057289592aa9f13ae8610b7768ca8e2e47ca1c9847325aa9480"} Jun 08 11:23:57.646034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:57.645787 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7" Jun 08 11:23:57.650593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:57.650571 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7" Jun 08 11:23:57.661057 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:57.660989 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-zsdh7" podStartSLOduration=1.588747185 podStartE2EDuration="2.660974404s" totalImagesPullingTime="1.072227219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:23:57.659795469 +0000 UTC m=+227.144194334" watchObservedRunningTime="2026-06-08 11:23:57.660974404 +0000 UTC m=+227.145373267" Jun 08 11:23:58.642354 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.642323 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p"] Jun 08 11:23:58.645758 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.645735 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.649426 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.649402 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jun 08 11:23:58.649570 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.649435 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jun 08 11:23:58.649570 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.649488 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jun 08 11:23:58.649570 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.649563 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jun 08 11:23:58.649779 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.649715 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4kblb\"" Jun 08 11:23:58.649839 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.649795 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jun 08 11:23:58.656266 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.656242 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p"] Jun 08 11:23:58.827002 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.826964 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlgc7\" (UniqueName: \"kubernetes.io/projected/e096e280-68d9-465a-8e52-562730844ab5-kube-api-access-rlgc7\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.827002 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.827004 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e096e280-68d9-465a-8e52-562730844ab5-metrics-client-ca\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.827258 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.827036 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e096e280-68d9-465a-8e52-562730844ab5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.827258 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.827127 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e096e280-68d9-465a-8e52-562730844ab5-prometheus-operator-tls\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.927688 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.927585 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlgc7\" (UniqueName: \"kubernetes.io/projected/e096e280-68d9-465a-8e52-562730844ab5-kube-api-access-rlgc7\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.927688 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.927655 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e096e280-68d9-465a-8e52-562730844ab5-metrics-client-ca\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.927952 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.927703 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e096e280-68d9-465a-8e52-562730844ab5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.927952 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.927770 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e096e280-68d9-465a-8e52-562730844ab5-prometheus-operator-tls\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.928484 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.928441 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e096e280-68d9-465a-8e52-562730844ab5-metrics-client-ca\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.930886 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.930858 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e096e280-68d9-465a-8e52-562730844ab5-prometheus-operator-tls\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.930886 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.930874 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e096e280-68d9-465a-8e52-562730844ab5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.935691 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.935669 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlgc7\" (UniqueName: \"kubernetes.io/projected/e096e280-68d9-465a-8e52-562730844ab5-kube-api-access-rlgc7\") pod \"prometheus-operator-7f7d445d84-gfg9p\" (UID: \"e096e280-68d9-465a-8e52-562730844ab5\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:58.958354 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:58.958316 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" Jun 08 11:23:59.102890 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:59.102841 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p"] Jun 08 11:23:59.105400 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:23:59.105371 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode096e280_68d9_465a_8e52_562730844ab5.slice/crio-9a5856eda6e8ea9b9b7f55c5d6f6740a0a35742043f7c2df98de6825547c4900 WatchSource:0}: Error finding container 9a5856eda6e8ea9b9b7f55c5d6f6740a0a35742043f7c2df98de6825547c4900: Status 404 returned error can't find the container with id 9a5856eda6e8ea9b9b7f55c5d6f6740a0a35742043f7c2df98de6825547c4900 Jun 08 11:23:59.654664 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:23:59.654573 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" event={"ID":"e096e280-68d9-465a-8e52-562730844ab5","Type":"ContainerStarted","Data":"9a5856eda6e8ea9b9b7f55c5d6f6740a0a35742043f7c2df98de6825547c4900"} Jun 08 11:24:00.661798 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:00.661695 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" event={"ID":"e096e280-68d9-465a-8e52-562730844ab5","Type":"ContainerStarted","Data":"c7835fa8ae39465161c408c429a3dfc605e6ad802ff00e2c99c38ca172475bc2"} Jun 08 11:24:00.661798 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:00.661770 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" event={"ID":"e096e280-68d9-465a-8e52-562730844ab5","Type":"ContainerStarted","Data":"1758e93dce50258b5906e1a102f2c22f105cd5edd196f9530d7054ebb9d11d49"} Jun 08 11:24:00.679320 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:00.679261 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7f7d445d84-gfg9p" podStartSLOduration=1.3741918229999999 podStartE2EDuration="2.679238875s" totalImagesPullingTime="1.305047052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:24:00.677900104 +0000 UTC m=+230.162298970" watchObservedRunningTime="2026-06-08 11:24:00.679238875 +0000 UTC m=+230.163637740" Jun 08 11:24:03.085896 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.085863 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-twz5m"] Jun 08 11:24:03.089436 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.089411 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.092019 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.091993 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jun 08 11:24:03.092565 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.092550 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rxs26\"" Jun 08 11:24:03.092779 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.092762 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jun 08 11:24:03.092947 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.092931 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jun 08 11:24:03.166969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.166934 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-accelerators-collector-config\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.167145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.166999 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-tls\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.167145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.167058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.167256 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.167151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0becfb22-31da-4d52-bd68-a80466b75ad0-metrics-client-ca\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.167256 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.167180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dpwp\" (UniqueName: \"kubernetes.io/projected/0becfb22-31da-4d52-bd68-a80466b75ad0-kube-api-access-9dpwp\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.167256 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.167218 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0becfb22-31da-4d52-bd68-a80466b75ad0-root\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.167256 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.167248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-wtmp\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.167446 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.167325 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0becfb22-31da-4d52-bd68-a80466b75ad0-sys\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.167446 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.167362 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-textfile\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.268583 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.268546 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0becfb22-31da-4d52-bd68-a80466b75ad0-metrics-client-ca\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.268583 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.268591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dpwp\" (UniqueName: \"kubernetes.io/projected/0becfb22-31da-4d52-bd68-a80466b75ad0-kube-api-access-9dpwp\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.268847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.268648 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0becfb22-31da-4d52-bd68-a80466b75ad0-root\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.268847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.268677 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-wtmp\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.268847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.268719 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0becfb22-31da-4d52-bd68-a80466b75ad0-sys\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.268847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.268748 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-textfile\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.268847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.268763 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0becfb22-31da-4d52-bd68-a80466b75ad0-root\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.268847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.268783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-accelerators-collector-config\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.269128 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.268858 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-tls\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.269128 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.268889 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.269413 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.269285 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0becfb22-31da-4d52-bd68-a80466b75ad0-sys\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.269413 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.269363 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0becfb22-31da-4d52-bd68-a80466b75ad0-metrics-client-ca\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.269546 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.269430 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-wtmp\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.269736 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.269687 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-accelerators-collector-config\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.270228 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.270166 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-textfile\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.272564 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.272539 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.272989 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.272966 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0becfb22-31da-4d52-bd68-a80466b75ad0-node-exporter-tls\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.277228 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.277202 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dpwp\" (UniqueName: \"kubernetes.io/projected/0becfb22-31da-4d52-bd68-a80466b75ad0-kube-api-access-9dpwp\") pod \"node-exporter-twz5m\" (UID: \"0becfb22-31da-4d52-bd68-a80466b75ad0\") " pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.398453 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.398362 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-twz5m" Jun 08 11:24:03.411239 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:24:03.411213 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0becfb22_31da_4d52_bd68_a80466b75ad0.slice/crio-c53504b33e8cbef19fa184df5e3883475aa45387dd01e1d1b1fb3e9273724a75 WatchSource:0}: Error finding container c53504b33e8cbef19fa184df5e3883475aa45387dd01e1d1b1fb3e9273724a75: Status 404 returned error can't find the container with id c53504b33e8cbef19fa184df5e3883475aa45387dd01e1d1b1fb3e9273724a75 Jun 08 11:24:03.672524 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:03.672437 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-twz5m" event={"ID":"0becfb22-31da-4d52-bd68-a80466b75ad0","Type":"ContainerStarted","Data":"c53504b33e8cbef19fa184df5e3883475aa45387dd01e1d1b1fb3e9273724a75"} Jun 08 11:24:04.114504 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.114471 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:24:04.119533 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.119505 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.122133 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.122110 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jun 08 11:24:04.122710 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.122653 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rpjz4\"" Jun 08 11:24:04.123021 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.122999 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jun 08 11:24:04.123219 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.123198 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jun 08 11:24:04.123301 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.123022 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jun 08 11:24:04.123301 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.123264 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jun 08 11:24:04.123301 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.123022 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jun 08 11:24:04.123455 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.123022 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jun 08 11:24:04.123455 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.123071 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jun 08 11:24:04.123455 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.123030 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jun 08 11:24:04.135008 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.134983 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:24:04.177435 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177393 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.177435 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177440 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.177716 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177467 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.177716 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.177716 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177650 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.177716 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177685 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-web-config\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.177904 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177768 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.177904 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vszhv\" (UniqueName: \"kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-kube-api-access-vszhv\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.177904 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177871 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.178036 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177900 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.178036 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177931 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-config-out\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.178036 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177959 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.178036 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.177988 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279053 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279014 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279053 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279057 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279324 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279324 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279147 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279324 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:24:04.279177 2569 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jun 08 11:24:04.279324 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:24:04.279264 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-main-tls podName:c4045f49-16b5-4a55-b397-ff8effe8f788 nodeName:}" failed. No retries permitted until 2026-06-08 11:24:04.779243337 +0000 UTC m=+234.263642179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788") : secret "alertmanager-main-tls" not found Jun 08 11:24:04.279324 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279290 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279324 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279321 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-web-config\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279521 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279370 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279521 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279432 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vszhv\" (UniqueName: \"kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-kube-api-access-vszhv\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279521 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279458 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279521 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279498 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279750 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279530 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-config-out\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279750 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.279750 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.279597 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.280265 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.280233 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.280444 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.280419 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.281146 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.280950 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.282390 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.282347 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.286742 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.283994 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.286742 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.284716 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.286742 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.284896 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-config-out\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.286742 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.284943 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-web-config\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.286742 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.285079 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.286742 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.285364 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.288315 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.288291 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.294282 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.294255 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vszhv\" (UniqueName: \"kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-kube-api-access-vszhv\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.678340 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.678248 2569 generic.go:356] "Generic (PLEG): container finished" podID="0becfb22-31da-4d52-bd68-a80466b75ad0" containerID="30575de04fb2cbd5db32709d31bd5eb5f0d81292f53aed975f71ccbabac782f4" exitCode=0 Jun 08 11:24:04.678498 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.678340 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-twz5m" event={"ID":"0becfb22-31da-4d52-bd68-a80466b75ad0","Type":"ContainerDied","Data":"30575de04fb2cbd5db32709d31bd5eb5f0d81292f53aed975f71ccbabac782f4"} Jun 08 11:24:04.784131 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.784089 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:04.786884 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:04.786861 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:05.031462 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:05.031413 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:24:06.032143 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.032106 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-694649bbf5-gvrsm"] Jun 08 11:24:06.036053 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.036029 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.038577 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.038512 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f0td4dikam1cm\"" Jun 08 11:24:06.038577 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.038571 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jun 08 11:24:06.038772 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.038631 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jun 08 11:24:06.038772 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.038705 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jun 08 11:24:06.038772 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.038751 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6qqsw\"" Jun 08 11:24:06.038772 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.038767 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jun 08 11:24:06.039526 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.039501 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jun 08 11:24:06.046222 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.046201 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-694649bbf5-gvrsm"] Jun 08 11:24:06.097763 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.097680 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.097763 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.097733 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-metrics-client-ca\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.097763 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.097764 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.098050 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.097792 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.098050 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.097842 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbwll\" (UniqueName: \"kubernetes.io/projected/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-kube-api-access-qbwll\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.098050 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.097920 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-grpc-tls\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.098194 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.098077 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.098245 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.098229 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-tls\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.199512 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.199469 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.199724 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.199589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-tls\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.199724 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.199637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.199724 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.199670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-metrics-client-ca\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.199724 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.199698 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.199928 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.199729 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.199928 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.199753 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbwll\" (UniqueName: \"kubernetes.io/projected/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-kube-api-access-qbwll\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.199928 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.199785 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-grpc-tls\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.200505 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.200456 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-metrics-client-ca\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.203064 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.203036 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.203064 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.203067 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.203258 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.203230 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.203317 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.203301 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-tls\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.204009 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.203986 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-grpc-tls\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.204262 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.204246 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.208014 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.207990 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbwll\" (UniqueName: \"kubernetes.io/projected/52a4c1ce-25bb-4193-bf2b-4f1322fa0d55-kube-api-access-qbwll\") pod \"thanos-querier-694649bbf5-gvrsm\" (UID: \"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55\") " pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:06.347860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:06.347771 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:07.361838 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.361800 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7547496f79-twqhk"] Jun 08 11:24:07.365256 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.365232 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.369968 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.369941 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jun 08 11:24:07.371248 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.371223 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6g63eik1qr4js\"" Jun 08 11:24:07.371415 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.371223 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jun 08 11:24:07.371415 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.371280 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jun 08 11:24:07.371415 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.371290 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jun 08 11:24:07.371600 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.371567 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dk5l5\"" Jun 08 11:24:07.392271 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.392241 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7547496f79-twqhk"] Jun 08 11:24:07.416375 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.416252 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/db77db9c-9324-4af1-b6fa-c01ed356156c-client-ca-bundle\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.416375 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.416363 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db77db9c-9324-4af1-b6fa-c01ed356156c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.416627 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.416466 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/db77db9c-9324-4af1-b6fa-c01ed356156c-metrics-server-audit-profiles\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.416627 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.416494 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6ff\" (UniqueName: \"kubernetes.io/projected/db77db9c-9324-4af1-b6fa-c01ed356156c-kube-api-access-8c6ff\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.416627 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.416534 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/db77db9c-9324-4af1-b6fa-c01ed356156c-secret-metrics-server-tls\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.416627 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.416564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/db77db9c-9324-4af1-b6fa-c01ed356156c-audit-log\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.416821 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.416671 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/db77db9c-9324-4af1-b6fa-c01ed356156c-secret-metrics-server-client-certs\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.518070 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.518032 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/db77db9c-9324-4af1-b6fa-c01ed356156c-secret-metrics-server-client-certs\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.518234 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.518084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/db77db9c-9324-4af1-b6fa-c01ed356156c-client-ca-bundle\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.518234 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.518115 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db77db9c-9324-4af1-b6fa-c01ed356156c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.518234 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.518179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/db77db9c-9324-4af1-b6fa-c01ed356156c-metrics-server-audit-profiles\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.518234 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.518201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6ff\" (UniqueName: \"kubernetes.io/projected/db77db9c-9324-4af1-b6fa-c01ed356156c-kube-api-access-8c6ff\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.518453 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.518241 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/db77db9c-9324-4af1-b6fa-c01ed356156c-secret-metrics-server-tls\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.518453 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.518266 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/db77db9c-9324-4af1-b6fa-c01ed356156c-audit-log\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.518996 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.518934 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/db77db9c-9324-4af1-b6fa-c01ed356156c-audit-log\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.519187 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.519137 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db77db9c-9324-4af1-b6fa-c01ed356156c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.519415 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.519368 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/db77db9c-9324-4af1-b6fa-c01ed356156c-metrics-server-audit-profiles\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.521001 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.520977 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/db77db9c-9324-4af1-b6fa-c01ed356156c-client-ca-bundle\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.521116 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.521092 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/db77db9c-9324-4af1-b6fa-c01ed356156c-secret-metrics-server-client-certs\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.521448 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.521424 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/db77db9c-9324-4af1-b6fa-c01ed356156c-secret-metrics-server-tls\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.526502 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.526476 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6ff\" (UniqueName: \"kubernetes.io/projected/db77db9c-9324-4af1-b6fa-c01ed356156c-kube-api-access-8c6ff\") pod \"metrics-server-7547496f79-twqhk\" (UID: \"db77db9c-9324-4af1-b6fa-c01ed356156c\") " pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:07.676825 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:07.676741 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:09.024155 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:09.024111 2569 patch_prober.go:28] interesting pod/image-registry-65949fcfbf-49m7t container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 08 11:24:09.026301 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:09.024189 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" podUID="177e0d64-9385-4ba2-a01f-d825fa7fddbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 08 11:24:10.617909 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:10.617859 2569 patch_prober.go:28] interesting pod/image-registry-65949fcfbf-49m7t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 08 11:24:10.618426 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:10.617924 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" podUID="177e0d64-9385-4ba2-a01f-d825fa7fddbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 08 11:24:11.899245 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:11.899141 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-694649bbf5-gvrsm"] Jun 08 11:24:11.908571 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:24:11.908543 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a4c1ce_25bb_4193_bf2b_4f1322fa0d55.slice/crio-4289f0614ab13f197c79c63efb59716e499b3779638bff10b5b78c428de30a49 WatchSource:0}: Error finding container 4289f0614ab13f197c79c63efb59716e499b3779638bff10b5b78c428de30a49: Status 404 returned error can't find the container with id 4289f0614ab13f197c79c63efb59716e499b3779638bff10b5b78c428de30a49 Jun 08 11:24:11.922138 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:11.922063 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:24:11.926005 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:24:11.925982 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4045f49_16b5_4a55_b397_ff8effe8f788.slice/crio-ffc2db6e586cfe743b68dbdb49a4fde66a413ac24beaa1184ff159748078a998 WatchSource:0}: Error finding container ffc2db6e586cfe743b68dbdb49a4fde66a413ac24beaa1184ff159748078a998: Status 404 returned error can't find the container with id ffc2db6e586cfe743b68dbdb49a4fde66a413ac24beaa1184ff159748078a998 Jun 08 11:24:12.118278 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:12.118201 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7547496f79-twqhk"] Jun 08 11:24:12.121647 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:24:12.121598 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb77db9c_9324_4af1_b6fa_c01ed356156c.slice/crio-fd9e89284611c537c19029d4238a2a9a60992c32611c68c56ffe264f1b61177c WatchSource:0}: Error finding container fd9e89284611c537c19029d4238a2a9a60992c32611c68c56ffe264f1b61177c: Status 404 returned error can't find the container with id fd9e89284611c537c19029d4238a2a9a60992c32611c68c56ffe264f1b61177c Jun 08 11:24:12.714880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:12.714838 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7547496f79-twqhk" event={"ID":"db77db9c-9324-4af1-b6fa-c01ed356156c","Type":"ContainerStarted","Data":"fd9e89284611c537c19029d4238a2a9a60992c32611c68c56ffe264f1b61177c"} Jun 08 11:24:12.717019 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:12.716980 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerStarted","Data":"ffc2db6e586cfe743b68dbdb49a4fde66a413ac24beaa1184ff159748078a998"} Jun 08 11:24:12.719926 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:12.719881 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" event={"ID":"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55","Type":"ContainerStarted","Data":"4289f0614ab13f197c79c63efb59716e499b3779638bff10b5b78c428de30a49"} Jun 08 11:24:12.725427 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:12.722533 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6657875f97-g6lcj" event={"ID":"66448ff5-45a3-4749-b993-e446af7d3dd6","Type":"ContainerStarted","Data":"348843e1b4cc0189ec98e10442e07f4b2bad05738972c1960a7862a3874eb56c"} Jun 08 11:24:12.725427 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:12.725237 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6657875f97-g6lcj" Jun 08 11:24:12.731558 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:12.730772 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-twz5m" event={"ID":"0becfb22-31da-4d52-bd68-a80466b75ad0","Type":"ContainerStarted","Data":"6aa3aeb00085f28635b4af7a47b4c1f268c2f7959a5a731d425837679b6e9fab"} Jun 08 11:24:12.731558 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:12.730827 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-twz5m" event={"ID":"0becfb22-31da-4d52-bd68-a80466b75ad0","Type":"ContainerStarted","Data":"e0045b757a71196fbf5713a4f7f42dab9a828006f9bff51247f0cec692c1fd9f"} Jun 08 11:24:12.737816 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:12.737748 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6657875f97-g6lcj" Jun 08 11:24:12.740566 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:12.740506 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6657875f97-g6lcj" podStartSLOduration=1.455608873 podStartE2EDuration="21.740488192s" totalImagesPullingTime="20.284879319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:24:12.739227007 +0000 UTC m=+242.223625870" watchObservedRunningTime="2026-06-08 11:24:12.740488192 +0000 UTC m=+242.224887053" Jun 08 11:24:13.738006 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:13.737967 2569 generic.go:356] "Generic (PLEG): container finished" podID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerID="9db0b05f8f5182c7563be8c9f105d57d30ea0cfb7d0708eeb14b78253dbe53ca" exitCode=0 Jun 08 11:24:13.738464 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:13.738122 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerDied","Data":"9db0b05f8f5182c7563be8c9f105d57d30ea0cfb7d0708eeb14b78253dbe53ca"} Jun 08 11:24:13.766175 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:13.766112 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-twz5m" podStartSLOduration=9.758711666 podStartE2EDuration="10.766089617s" totalImagesPullingTime="921.411426ms" totalInitContainerRuntime="85.966525ms" isStatefulPod=true podCreationTimestamp="2026-06-08 11:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:24:12.773972837 +0000 UTC m=+242.258371701" watchObservedRunningTime="2026-06-08 11:24:13.766089617 +0000 UTC m=+243.250488482" Jun 08 11:24:15.751098 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:15.751052 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7547496f79-twqhk" event={"ID":"db77db9c-9324-4af1-b6fa-c01ed356156c","Type":"ContainerStarted","Data":"17b142d7f7688a3a05d7cfd768af7402b112554104230debdb0a83e16f0ceb47"} Jun 08 11:24:15.754709 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:15.754653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" event={"ID":"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55","Type":"ContainerStarted","Data":"598aefebb71281851ef2da07879c0bbf47eea569863ee981726600c008ef12c1"} Jun 08 11:24:15.754846 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:15.754734 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" event={"ID":"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55","Type":"ContainerStarted","Data":"bc3b1e3af87e6fe875eb138f137b7c3e1527fd629876a190048c398dd08c7892"} Jun 08 11:24:15.754846 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:15.754796 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" event={"ID":"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55","Type":"ContainerStarted","Data":"aebdaa38f98b70cd78def08a011922376ba803068eca7d3040d39931b96f22a2"} Jun 08 11:24:15.769430 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:15.769364 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7547496f79-twqhk" podStartSLOduration=5.72330254 podStartE2EDuration="8.769344332s" totalImagesPullingTime="3.046041792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:24:15.767168909 +0000 UTC m=+245.251567773" watchObservedRunningTime="2026-06-08 11:24:15.769344332 +0000 UTC m=+245.253743195" Jun 08 11:24:17.768587 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:17.768526 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" event={"ID":"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55","Type":"ContainerStarted","Data":"3fde65f4cefda02a51f5e9d04d8bad045a624f8a4686106adcd851e5bd4ee51c"} Jun 08 11:24:17.782561 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:17.782481 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerStarted","Data":"ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072"} Jun 08 11:24:17.782758 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:17.782586 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerStarted","Data":"a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40"} Jun 08 11:24:17.782758 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:17.782648 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerStarted","Data":"5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3"} Jun 08 11:24:17.782758 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:17.782691 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerStarted","Data":"4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97"} Jun 08 11:24:17.782758 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:17.782731 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerStarted","Data":"ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5"} Jun 08 11:24:18.790776 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:18.790737 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerStarted","Data":"b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292"} Jun 08 11:24:18.794363 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:18.794325 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" event={"ID":"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55","Type":"ContainerStarted","Data":"3aefd696e4aaf6bfdde0d64ce80411cc27cfa239f83ac346c2c8d53b44311176"} Jun 08 11:24:18.794514 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:18.794400 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" event={"ID":"52a4c1ce-25bb-4193-bf2b-4f1322fa0d55","Type":"ContainerStarted","Data":"360eb9befe7b6e970c2cdd3e7aa144827f632df01b11fa665914d5f249f474af"} Jun 08 11:24:18.794514 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:18.794446 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:18.818835 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:18.818775 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=10.53020145 podStartE2EDuration="14.818754338s" totalImagesPullingTime="4.263327502s" totalInitContainerRuntime="25.225386ms" isStatefulPod=false podCreationTimestamp="2026-06-08 11:24:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:24:18.817675146 +0000 UTC m=+248.302074009" watchObservedRunningTime="2026-06-08 11:24:18.818754338 +0000 UTC m=+248.303153207" Jun 08 11:24:19.024542 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:19.024502 2569 patch_prober.go:28] interesting pod/image-registry-65949fcfbf-49m7t container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 08 11:24:19.024742 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:19.024565 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" podUID="177e0d64-9385-4ba2-a01f-d825fa7fddbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 08 11:24:20.617923 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:20.617886 2569 patch_prober.go:28] interesting pod/image-registry-65949fcfbf-49m7t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 08 11:24:20.618395 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:20.617941 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" podUID="177e0d64-9385-4ba2-a01f-d825fa7fddbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 08 11:24:22.770760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:22.770715 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:24:22.773237 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:22.773209 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2413fc4e-6ee1-47a8-8621-657e43ccfcf4-metrics-certs\") pod \"network-metrics-daemon-kthbk\" (UID: \"2413fc4e-6ee1-47a8-8621-657e43ccfcf4\") " pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:24:22.851910 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:22.851879 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6vkk7\"" Jun 08 11:24:22.859821 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:22.859797 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kthbk" Jun 08 11:24:23.007259 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:23.007199 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" podStartSLOduration=11.840089899 podStartE2EDuration="17.007179465s" totalImagesPullingTime="5.167089566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:24:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:24:18.845261977 +0000 UTC m=+248.329660844" watchObservedRunningTime="2026-06-08 11:24:23.007179465 +0000 UTC m=+252.491578310" Jun 08 11:24:23.009293 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:23.009265 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kthbk"] Jun 08 11:24:23.012323 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:24:23.012292 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2413fc4e_6ee1_47a8_8621_657e43ccfcf4.slice/crio-0d825d11d959fe430dee374158114a8877296672b2974fecf9f7c364426db8e4 WatchSource:0}: Error finding container 0d825d11d959fe430dee374158114a8877296672b2974fecf9f7c364426db8e4: Status 404 returned error can't find the container with id 0d825d11d959fe430dee374158114a8877296672b2974fecf9f7c364426db8e4 Jun 08 11:24:23.816425 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:23.816385 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kthbk" event={"ID":"2413fc4e-6ee1-47a8-8621-657e43ccfcf4","Type":"ContainerStarted","Data":"0d825d11d959fe430dee374158114a8877296672b2974fecf9f7c364426db8e4"} Jun 08 11:24:24.805767 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:24.805742 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-694649bbf5-gvrsm" Jun 08 11:24:24.821828 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:24.821801 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kthbk" event={"ID":"2413fc4e-6ee1-47a8-8621-657e43ccfcf4","Type":"ContainerStarted","Data":"7e29824eab47e63de1e905af19a10b6af0fc79d4625670c631c6327273113e24"} Jun 08 11:24:24.822147 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:24.821849 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kthbk" event={"ID":"2413fc4e-6ee1-47a8-8621-657e43ccfcf4","Type":"ContainerStarted","Data":"0f9ada7653f98fb1880735383fead5894c02f1cd02085db2e26f26462564ba62"} Jun 08 11:24:25.842166 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:25.842104 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kthbk" podStartSLOduration=253.322249523 podStartE2EDuration="4m14.842086233s" totalImagesPullingTime="1.51983671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:24:25.840606428 +0000 UTC m=+255.325005303" watchObservedRunningTime="2026-06-08 11:24:25.842086233 +0000 UTC m=+255.326485096" Jun 08 11:24:27.677108 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:27.677069 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:27.677506 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:27.677129 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:29.024761 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:29.024720 2569 patch_prober.go:28] interesting pod/image-registry-65949fcfbf-49m7t container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 08 11:24:29.025128 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:29.024773 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" podUID="177e0d64-9385-4ba2-a01f-d825fa7fddbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 08 11:24:29.025128 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:29.024812 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:24:29.025380 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:29.025323 2569 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"b86b3a4280e7c19b27dee73e206e23ec7a8bcccb011a741f8c05a969d35e24cb"} pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" containerMessage="Container registry failed liveness probe, will be restarted" Jun 08 11:24:29.029212 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:29.029171 2569 patch_prober.go:28] interesting pod/image-registry-65949fcfbf-49m7t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 08 11:24:29.029350 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:29.029234 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" podUID="177e0d64-9385-4ba2-a01f-d825fa7fddbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 08 11:24:29.842425 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:29.842391 2569 generic.go:356] "Generic (PLEG): container finished" podID="fc8847f9-8e90-498b-9c06-237949ccad50" containerID="b8965e2d05173cd48acf44bc5d58ac6483fd274b0206698999aab542724eee01" exitCode=0 Jun 08 11:24:29.842746 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:29.842459 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-8l62d" event={"ID":"fc8847f9-8e90-498b-9c06-237949ccad50","Type":"ContainerDied","Data":"b8965e2d05173cd48acf44bc5d58ac6483fd274b0206698999aab542724eee01"} Jun 08 11:24:29.842874 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:29.842856 2569 scope.go:117] "RemoveContainer" containerID="b8965e2d05173cd48acf44bc5d58ac6483fd274b0206698999aab542724eee01" Jun 08 11:24:30.848434 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:30.848402 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-8l62d" event={"ID":"fc8847f9-8e90-498b-9c06-237949ccad50","Type":"ContainerStarted","Data":"b62fe6bdceeb2435e68cb669763d9046f747cd1f46304091de89068107bedc88"} Jun 08 11:24:32.999649 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:32.999585 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4045f49-16b5-4a55-b397-ff8effe8f788/init-config-reloader/0.log" Jun 08 11:24:33.199305 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:33.199276 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4045f49-16b5-4a55-b397-ff8effe8f788/alertmanager/0.log" Jun 08 11:24:33.399808 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:33.399714 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4045f49-16b5-4a55-b397-ff8effe8f788/config-reloader/0.log" Jun 08 11:24:33.599428 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:33.599376 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4045f49-16b5-4a55-b397-ff8effe8f788/kube-rbac-proxy-web/0.log" Jun 08 11:24:33.799678 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:33.799653 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4045f49-16b5-4a55-b397-ff8effe8f788/kube-rbac-proxy/0.log" Jun 08 11:24:33.999351 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:33.999314 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4045f49-16b5-4a55-b397-ff8effe8f788/kube-rbac-proxy-metric/0.log" Jun 08 11:24:34.199604 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:34.199533 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4045f49-16b5-4a55-b397-ff8effe8f788/prom-label-proxy/0.log" Jun 08 11:24:35.198908 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:35.198877 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7547496f79-twqhk_db77db9c-9324-4af1-b6fa-c01ed356156c/metrics-server/0.log" Jun 08 11:24:36.798475 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:36.798447 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-twz5m_0becfb22-31da-4d52-bd68-a80466b75ad0/init-textfile/0.log" Jun 08 11:24:36.999338 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:36.999312 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-twz5m_0becfb22-31da-4d52-bd68-a80466b75ad0/node-exporter/0.log" Jun 08 11:24:37.199554 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:37.199464 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-twz5m_0becfb22-31da-4d52-bd68-a80466b75ad0/kube-rbac-proxy/0.log" Jun 08 11:24:39.029221 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:39.029179 2569 patch_prober.go:28] interesting pod/image-registry-65949fcfbf-49m7t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 08 11:24:39.029596 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:39.029229 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" podUID="177e0d64-9385-4ba2-a01f-d825fa7fddbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 08 11:24:39.400648 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:39.400559 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7f7d445d84-gfg9p_e096e280-68d9-465a-8e52-562730844ab5/prometheus-operator/0.log" Jun 08 11:24:39.598685 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:39.598643 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7f7d445d84-gfg9p_e096e280-68d9-465a-8e52-562730844ab5/kube-rbac-proxy/0.log" Jun 08 11:24:39.799239 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:39.799206 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5b6b8f594-zsdh7_c74c35a7-f89f-45b9-b8c0-ca6f2f91c562/prometheus-operator-admission-webhook/0.log" Jun 08 11:24:39.999098 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:39.999066 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/thanos-query/0.log" Jun 08 11:24:40.199193 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:40.199116 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/kube-rbac-proxy-web/0.log" Jun 08 11:24:40.399712 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:40.399685 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/kube-rbac-proxy/0.log" Jun 08 11:24:40.599427 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:40.599393 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/prom-label-proxy/0.log" Jun 08 11:24:40.799767 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:40.799741 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/kube-rbac-proxy-rules/0.log" Jun 08 11:24:40.999234 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:40.999209 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/kube-rbac-proxy-metrics/0.log" Jun 08 11:24:41.199783 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:41.199746 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d5d5bb78b-cnm4d_ef8c72d1-5ce7-44c0-a259-cf8b40983674/networking-console-plugin/0.log" Jun 08 11:24:42.201193 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:42.201162 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6657875f97-g6lcj_66448ff5-45a3-4749-b993-e446af7d3dd6/download-server/0.log" Jun 08 11:24:43.999236 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:43.999206 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bkxdh_822bda63-ed87-40ff-86a1-5ebee377ed5a/dns-node-resolver/0.log" Jun 08 11:24:44.200497 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:44.200458 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65949fcfbf-49m7t_177e0d64-9385-4ba2-a01f-d825fa7fddbf/registry/0.log" Jun 08 11:24:44.999404 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:44.999378 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zmxqf_25861972-bb71-4dc8-bfae-31d8c5a5bd8e/node-ca/0.log" Jun 08 11:24:45.199956 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:45.199925 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f975c8c69-c85sm_2bcc00d8-28dc-4f53-ba05-4b53c852e037/router/0.log" Jun 08 11:24:45.799418 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:45.799394 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jjjwt_74ad6f9e-7b62-45d8-80b6-391b0f3758cb/serve-healthcheck-canary/0.log" Jun 08 11:24:47.686940 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:47.686912 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:47.690879 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:47.690848 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7547496f79-twqhk" Jun 08 11:24:49.030436 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:49.030402 2569 patch_prober.go:28] interesting pod/image-registry-65949fcfbf-49m7t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 08 11:24:49.030832 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:49.030471 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" podUID="177e0d64-9385-4ba2-a01f-d825fa7fddbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 08 11:24:54.045176 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:54.045123 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" podUID="177e0d64-9385-4ba2-a01f-d825fa7fddbf" containerName="registry" containerID="cri-o://b86b3a4280e7c19b27dee73e206e23ec7a8bcccb011a741f8c05a969d35e24cb" gracePeriod=30 Jun 08 11:24:55.865785 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:55.865739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:24:55.868214 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:55.868194 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/483118b0-02a0-4b58-baf7-2e16a4fb6e2e-metrics-tls\") pod \"dns-default-n5m8b\" (UID: \"483118b0-02a0-4b58-baf7-2e16a4fb6e2e\") " pod="openshift-dns/dns-default-n5m8b" Jun 08 11:24:55.970949 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:55.970917 2569 generic.go:356] "Generic (PLEG): container finished" podID="177e0d64-9385-4ba2-a01f-d825fa7fddbf" containerID="b86b3a4280e7c19b27dee73e206e23ec7a8bcccb011a741f8c05a969d35e24cb" exitCode=0 Jun 08 11:24:55.971115 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:55.970970 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" event={"ID":"177e0d64-9385-4ba2-a01f-d825fa7fddbf","Type":"ContainerDied","Data":"b86b3a4280e7c19b27dee73e206e23ec7a8bcccb011a741f8c05a969d35e24cb"} Jun 08 11:24:55.971115 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:55.971017 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" event={"ID":"177e0d64-9385-4ba2-a01f-d825fa7fddbf","Type":"ContainerStarted","Data":"686ea95ce559258e81e9930fd41c37a7029c7e515a1031ead5e88d2d54dadf38"} Jun 08 11:24:55.971227 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:55.971202 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:24:56.152901 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:56.152818 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-flcnh\"" Jun 08 11:24:56.160692 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:56.160668 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n5m8b" Jun 08 11:24:56.327839 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:56.327812 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n5m8b"] Jun 08 11:24:56.330569 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:24:56.330538 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483118b0_02a0_4b58_baf7_2e16a4fb6e2e.slice/crio-a7cda6a5bc2199990382ab304dd3fe2a20efd17d37b73a265d2c55b674e51737 WatchSource:0}: Error finding container a7cda6a5bc2199990382ab304dd3fe2a20efd17d37b73a265d2c55b674e51737: Status 404 returned error can't find the container with id a7cda6a5bc2199990382ab304dd3fe2a20efd17d37b73a265d2c55b674e51737 Jun 08 11:24:56.977272 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:56.977228 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n5m8b" event={"ID":"483118b0-02a0-4b58-baf7-2e16a4fb6e2e","Type":"ContainerStarted","Data":"a7cda6a5bc2199990382ab304dd3fe2a20efd17d37b73a265d2c55b674e51737"} Jun 08 11:24:58.986798 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:58.986761 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n5m8b" event={"ID":"483118b0-02a0-4b58-baf7-2e16a4fb6e2e","Type":"ContainerStarted","Data":"a67cf981ca23437f33a9e8fa6cde979aa90bd6743d67592118abee527f88e617"} Jun 08 11:24:58.987245 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:58.986823 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n5m8b" event={"ID":"483118b0-02a0-4b58-baf7-2e16a4fb6e2e","Type":"ContainerStarted","Data":"a6b369ceabcd6b6d2b985d8611f6f1c4faad41650c0331d82818f760962dcb17"} Jun 08 11:24:58.987245 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:58.986908 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n5m8b" Jun 08 11:24:59.003965 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:24:59.003917 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n5m8b" podStartSLOduration=251.983578892 podStartE2EDuration="4m14.00389969s" totalImagesPullingTime="2.020320798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:24:59.003255774 +0000 UTC m=+288.487654640" watchObservedRunningTime="2026-06-08 11:24:59.00389969 +0000 UTC m=+288.488298557" Jun 08 11:25:08.992664 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:08.992633 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n5m8b" Jun 08 11:25:10.903031 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:10.903003 2569 kubelet.go:1648] "Image garbage collection succeeded" Jun 08 11:25:16.981575 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:16.981548 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65949fcfbf-49m7t" Jun 08 11:25:23.230871 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:23.230840 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:25:23.231253 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:23.231092 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="alertmanager" containerID="cri-o://ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5" gracePeriod=120 Jun 08 11:25:23.231253 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:23.231171 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy-web" containerID="cri-o://5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3" gracePeriod=120 Jun 08 11:25:23.231253 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:23.231166 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy-metric" containerID="cri-o://ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072" gracePeriod=120 Jun 08 11:25:23.231253 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:23.231226 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="prom-label-proxy" containerID="cri-o://b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292" gracePeriod=120 Jun 08 11:25:23.231461 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:23.231244 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy" containerID="cri-o://a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40" gracePeriod=120 Jun 08 11:25:23.231461 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:23.231185 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="config-reloader" containerID="cri-o://4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97" gracePeriod=120 Jun 08 11:25:24.090704 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.090670 2569 generic.go:356] "Generic (PLEG): container finished" podID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerID="b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292" exitCode=0 Jun 08 11:25:24.090704 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.090695 2569 generic.go:356] "Generic (PLEG): container finished" podID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerID="ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072" exitCode=0 Jun 08 11:25:24.090704 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.090703 2569 generic.go:356] "Generic (PLEG): container finished" podID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerID="a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40" exitCode=0 Jun 08 11:25:24.090704 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.090709 2569 generic.go:356] "Generic (PLEG): container finished" podID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerID="4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97" exitCode=0 Jun 08 11:25:24.090704 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.090715 2569 generic.go:356] "Generic (PLEG): container finished" podID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerID="ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5" exitCode=0 Jun 08 11:25:24.090979 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.090739 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerDied","Data":"b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292"} Jun 08 11:25:24.090979 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.090800 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerDied","Data":"ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072"} Jun 08 11:25:24.090979 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.090826 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerDied","Data":"a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40"} Jun 08 11:25:24.090979 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.090849 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerDied","Data":"4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97"} Jun 08 11:25:24.090979 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.090871 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerDied","Data":"ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5"} Jun 08 11:25:24.479829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.479805 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:24.618648 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.618512 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-config-out\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.618648 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.618601 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-main-tls\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.618648 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.618648 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.618867 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.618669 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-metrics-client-ca\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.618867 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.618708 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.618867 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.618726 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-web-config\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.619072 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.619042 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-cluster-tls-config\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.619222 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.619089 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-trusted-ca-bundle\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.619222 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.619115 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:25:24.619222 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.619126 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-config-volume\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.619222 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.619219 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-tls-assets\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.619448 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.619253 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-main-db\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.619448 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.619287 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vszhv\" (UniqueName: \"kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-kube-api-access-vszhv\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.619747 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.619466 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-web\") pod \"c4045f49-16b5-4a55-b397-ff8effe8f788\" (UID: \"c4045f49-16b5-4a55-b397-ff8effe8f788\") " Jun 08 11:25:24.619812 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.619800 2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-metrics-client-ca\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.621582 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.621550 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:25:24.622122 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.621887 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-config-out" (OuterVolumeSpecName: "config-out") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:25:24.622122 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.622042 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:25:24.622260 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.622212 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-config-volume" (OuterVolumeSpecName: "config-volume") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:25:24.622260 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.622242 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:25:24.622351 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.622259 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:25:24.622351 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.622319 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:25:24.622446 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.622379 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:25:24.624107 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.624082 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-kube-api-access-vszhv" (OuterVolumeSpecName: "kube-api-access-vszhv") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "kube-api-access-vszhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:25:24.624383 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.624363 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:25:24.627275 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.627253 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:25:24.632943 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.632921 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-web-config" (OuterVolumeSpecName: "web-config") pod "c4045f49-16b5-4a55-b397-ff8effe8f788" (UID: "c4045f49-16b5-4a55-b397-ff8effe8f788"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:25:24.720253 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720199 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.720253 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720248 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-web-config\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.720253 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720263 2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-cluster-tls-config\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.720520 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720277 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.720520 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720289 2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-config-volume\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.720520 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720300 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-tls-assets\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.720520 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720310 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-alertmanager-main-db\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.720520 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720320 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vszhv\" (UniqueName: \"kubernetes.io/projected/c4045f49-16b5-4a55-b397-ff8effe8f788-kube-api-access-vszhv\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.720520 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720333 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.720520 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720360 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4045f49-16b5-4a55-b397-ff8effe8f788-config-out\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.720520 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720373 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-main-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:24.720520 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:24.720385 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4045f49-16b5-4a55-b397-ff8effe8f788-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:25:25.097110 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.097076 2569 generic.go:356] "Generic (PLEG): container finished" podID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerID="5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3" exitCode=0 Jun 08 11:25:25.097277 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.097112 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerDied","Data":"5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3"} Jun 08 11:25:25.097277 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.097193 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.097277 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.097200 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4045f49-16b5-4a55-b397-ff8effe8f788","Type":"ContainerDied","Data":"ffc2db6e586cfe743b68dbdb49a4fde66a413ac24beaa1184ff159748078a998"} Jun 08 11:25:25.097277 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.097248 2569 scope.go:117] "RemoveContainer" containerID="b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292" Jun 08 11:25:25.105629 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.105592 2569 scope.go:117] "RemoveContainer" containerID="ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072" Jun 08 11:25:25.112687 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.112671 2569 scope.go:117] "RemoveContainer" containerID="a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40" Jun 08 11:25:25.116932 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.116906 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:25:25.120930 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.120906 2569 scope.go:117] "RemoveContainer" containerID="5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3" Jun 08 11:25:25.121023 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.120942 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:25:25.128011 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.127994 2569 scope.go:117] "RemoveContainer" containerID="4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97" Jun 08 11:25:25.134707 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.134688 2569 scope.go:117] "RemoveContainer" containerID="ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5" Jun 08 11:25:25.141761 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.141730 2569 scope.go:117] "RemoveContainer" containerID="9db0b05f8f5182c7563be8c9f105d57d30ea0cfb7d0708eeb14b78253dbe53ca" Jun 08 11:25:25.145654 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.145632 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:25:25.145962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.145943 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="config-reloader" Jun 08 11:25:25.145962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.145962 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="config-reloader" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.145973 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="alertmanager" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.145980 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="alertmanager" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146000 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy-web" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146006 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy-web" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146019 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="prom-label-proxy" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146024 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="prom-label-proxy" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146031 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="init-config-reloader" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146036 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="init-config-reloader" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146043 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy-metric" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146048 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy-metric" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146056 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy" Jun 08 11:25:25.146094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146062 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy" Jun 08 11:25:25.146445 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146123 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="alertmanager" Jun 08 11:25:25.146445 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146131 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="config-reloader" Jun 08 11:25:25.146445 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146139 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy" Jun 08 11:25:25.146445 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146147 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy-web" Jun 08 11:25:25.146445 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146157 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="kube-rbac-proxy-metric" Jun 08 11:25:25.146445 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.146164 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" containerName="prom-label-proxy" Jun 08 11:25:25.149386 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.149367 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.150008 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.149988 2569 scope.go:117] "RemoveContainer" containerID="b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292" Jun 08 11:25:25.150346 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:25:25.150314 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292\": container with ID starting with b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292 not found: ID does not exist" containerID="b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292" Jun 08 11:25:25.150435 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.150358 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292"} err="failed to get container status \"b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292\": rpc error: code = NotFound desc = could not find container \"b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292\": container with ID starting with b8596b311f4cf61c36de633e4eb0e779d3a7e118df65d3266c7e1edbbf248292 not found: ID does not exist" Jun 08 11:25:25.150435 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.150383 2569 scope.go:117] "RemoveContainer" containerID="ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072" Jun 08 11:25:25.150726 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:25:25.150694 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072\": container with ID starting with ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072 not found: ID does not exist" containerID="ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072" Jun 08 11:25:25.150841 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.150730 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072"} err="failed to get container status \"ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072\": rpc error: code = NotFound desc = could not find container \"ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072\": container with ID starting with ff08b60aea4a5e46ea4771c755b87b74797aef926ed6434c279804102b991072 not found: ID does not exist" Jun 08 11:25:25.150841 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.150748 2569 scope.go:117] "RemoveContainer" containerID="a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40" Jun 08 11:25:25.151011 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:25:25.150995 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40\": container with ID starting with a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40 not found: ID does not exist" containerID="a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40" Jun 08 11:25:25.151054 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.151017 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40"} err="failed to get container status \"a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40\": rpc error: code = NotFound desc = could not find container \"a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40\": container with ID starting with a3063acd0be71e70181a0da9b7fbecc991cdb7dd826e856811da92d09e814a40 not found: ID does not exist" Jun 08 11:25:25.151054 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.151031 2569 scope.go:117] "RemoveContainer" containerID="5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3" Jun 08 11:25:25.151295 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:25:25.151275 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3\": container with ID starting with 5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3 not found: ID does not exist" containerID="5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3" Jun 08 11:25:25.151366 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.151298 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3"} err="failed to get container status \"5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3\": rpc error: code = NotFound desc = could not find container \"5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3\": container with ID starting with 5ea0e86cf763946b9747ddfcd27789b2ae45ab6781eee71d7d5b39cd1c26f1f3 not found: ID does not exist" Jun 08 11:25:25.151366 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.151312 2569 scope.go:117] "RemoveContainer" containerID="4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97" Jun 08 11:25:25.151593 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:25:25.151560 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97\": container with ID starting with 4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97 not found: ID does not exist" containerID="4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97" Jun 08 11:25:25.151718 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.151587 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97"} err="failed to get container status \"4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97\": rpc error: code = NotFound desc = could not find container \"4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97\": container with ID starting with 4456a0a2617283a5c5e9469bce81a3ff06830442fc6fdc6d2915254489fcea97 not found: ID does not exist" Jun 08 11:25:25.151718 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.151630 2569 scope.go:117] "RemoveContainer" containerID="ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5" Jun 08 11:25:25.151850 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.151729 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jun 08 11:25:25.151850 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.151783 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jun 08 11:25:25.151850 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.151797 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jun 08 11:25:25.151850 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.151834 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jun 08 11:25:25.152170 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.151789 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jun 08 11:25:25.152170 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:25:25.152000 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5\": container with ID starting with ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5 not found: ID does not exist" containerID="ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5" Jun 08 11:25:25.152170 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.152027 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5"} err="failed to get container status \"ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5\": rpc error: code = NotFound desc = could not find container \"ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5\": container with ID starting with ed73bdc03f510c9843fdad25f161579f7ae2b3024b6b8ffab9bc57076f7355c5 not found: ID does not exist" Jun 08 11:25:25.152170 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.152049 2569 scope.go:117] "RemoveContainer" containerID="9db0b05f8f5182c7563be8c9f105d57d30ea0cfb7d0708eeb14b78253dbe53ca" Jun 08 11:25:25.152170 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.152141 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jun 08 11:25:25.152170 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.152158 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rpjz4\"" Jun 08 11:25:25.152598 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.152180 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jun 08 11:25:25.152598 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.152229 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jun 08 11:25:25.152598 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:25:25.152306 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db0b05f8f5182c7563be8c9f105d57d30ea0cfb7d0708eeb14b78253dbe53ca\": container with ID starting with 9db0b05f8f5182c7563be8c9f105d57d30ea0cfb7d0708eeb14b78253dbe53ca not found: ID does not exist" containerID="9db0b05f8f5182c7563be8c9f105d57d30ea0cfb7d0708eeb14b78253dbe53ca" Jun 08 11:25:25.152598 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.152329 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db0b05f8f5182c7563be8c9f105d57d30ea0cfb7d0708eeb14b78253dbe53ca"} err="failed to get container status \"9db0b05f8f5182c7563be8c9f105d57d30ea0cfb7d0708eeb14b78253dbe53ca\": rpc error: code = NotFound desc = could not find container \"9db0b05f8f5182c7563be8c9f105d57d30ea0cfb7d0708eeb14b78253dbe53ca\": container with ID starting with 9db0b05f8f5182c7563be8c9f105d57d30ea0cfb7d0708eeb14b78253dbe53ca not found: ID does not exist" Jun 08 11:25:25.157411 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.157392 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jun 08 11:25:25.163215 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.163193 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:25:25.324988 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.324958 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djv9s\" (UniqueName: \"kubernetes.io/projected/a7302a48-bc54-494e-bdea-b6f958088c7b-kube-api-access-djv9s\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325109 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.324993 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7302a48-bc54-494e-bdea-b6f958088c7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325109 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.325012 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325109 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.325032 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325109 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.325075 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.325118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7302a48-bc54-494e-bdea-b6f958088c7b-config-out\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.325157 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7302a48-bc54-494e-bdea-b6f958088c7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.325178 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.325219 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a7302a48-bc54-494e-bdea-b6f958088c7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325345 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.325256 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325345 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.325302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325345 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.325325 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-web-config\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.325476 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.325348 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7302a48-bc54-494e-bdea-b6f958088c7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426107 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426074 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7302a48-bc54-494e-bdea-b6f958088c7b-config-out\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426107 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426109 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7302a48-bc54-494e-bdea-b6f958088c7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426295 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426128 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426295 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426265 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a7302a48-bc54-494e-bdea-b6f958088c7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426390 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426312 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426486 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426461 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426536 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-web-config\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426588 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426540 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7302a48-bc54-494e-bdea-b6f958088c7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426691 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426671 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a7302a48-bc54-494e-bdea-b6f958088c7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426747 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426731 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djv9s\" (UniqueName: \"kubernetes.io/projected/a7302a48-bc54-494e-bdea-b6f958088c7b-kube-api-access-djv9s\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426801 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426765 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7302a48-bc54-494e-bdea-b6f958088c7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426801 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426900 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426823 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.426900 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.426851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.427313 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.427232 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7302a48-bc54-494e-bdea-b6f958088c7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.427578 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.427390 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7302a48-bc54-494e-bdea-b6f958088c7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.429626 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.429545 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.429626 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.429553 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.429626 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.429594 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7302a48-bc54-494e-bdea-b6f958088c7b-config-out\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.429850 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.429738 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.429850 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.429774 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.429984 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.429958 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7302a48-bc54-494e-bdea-b6f958088c7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.430691 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.430668 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.430779 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.430727 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-web-config\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.431797 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.431779 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a7302a48-bc54-494e-bdea-b6f958088c7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.435311 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.435294 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djv9s\" (UniqueName: \"kubernetes.io/projected/a7302a48-bc54-494e-bdea-b6f958088c7b-kube-api-access-djv9s\") pod \"alertmanager-main-0\" (UID: \"a7302a48-bc54-494e-bdea-b6f958088c7b\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.459386 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.459359 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:25:25.599997 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:25.599785 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:25:25.602473 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:25:25.602442 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7302a48_bc54_494e_bdea_b6f958088c7b.slice/crio-173fb1aa2080e862bc43280ba487428d97c44141d156534b102edd8fe487ea73 WatchSource:0}: Error finding container 173fb1aa2080e862bc43280ba487428d97c44141d156534b102edd8fe487ea73: Status 404 returned error can't find the container with id 173fb1aa2080e862bc43280ba487428d97c44141d156534b102edd8fe487ea73 Jun 08 11:25:26.103201 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:26.103167 2569 generic.go:356] "Generic (PLEG): container finished" podID="a7302a48-bc54-494e-bdea-b6f958088c7b" containerID="23202a29fcfc2988137923a08aa3acd32bb5ed7da57f0997b7f7bdeb7426b885" exitCode=0 Jun 08 11:25:26.103378 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:26.103233 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7302a48-bc54-494e-bdea-b6f958088c7b","Type":"ContainerDied","Data":"23202a29fcfc2988137923a08aa3acd32bb5ed7da57f0997b7f7bdeb7426b885"} Jun 08 11:25:26.103378 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:26.103279 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7302a48-bc54-494e-bdea-b6f958088c7b","Type":"ContainerStarted","Data":"173fb1aa2080e862bc43280ba487428d97c44141d156534b102edd8fe487ea73"} Jun 08 11:25:26.954501 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:26.954467 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4045f49-16b5-4a55-b397-ff8effe8f788" path="/var/lib/kubelet/pods/c4045f49-16b5-4a55-b397-ff8effe8f788/volumes" Jun 08 11:25:27.111109 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.111076 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7302a48-bc54-494e-bdea-b6f958088c7b","Type":"ContainerStarted","Data":"8e52a7e1687b8c39b84c76c3eec2f3522da84024a5f4d9b626b696a3f8e58fc3"} Jun 08 11:25:27.111258 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.111141 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7302a48-bc54-494e-bdea-b6f958088c7b","Type":"ContainerStarted","Data":"7b8f7b2dadd70a3ca32b627fcd2ac9e5adec142082c968b25eec2a1333754d71"} Jun 08 11:25:27.111258 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.111171 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7302a48-bc54-494e-bdea-b6f958088c7b","Type":"ContainerStarted","Data":"99dc18ffaedecd5c1d89ef3bb9ad094d679093e7aa59dd7f313628899c7ae503"} Jun 08 11:25:27.111258 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.111196 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7302a48-bc54-494e-bdea-b6f958088c7b","Type":"ContainerStarted","Data":"587a4fe9601bfd0e82ec8ae1739cb72f523360a4448080d349bed9287bcf4da2"} Jun 08 11:25:27.111258 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.111219 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7302a48-bc54-494e-bdea-b6f958088c7b","Type":"ContainerStarted","Data":"b00e3451e2c2c8f0c89e99839ee19db13c0aed1de81d750601a70621eafca236"} Jun 08 11:25:27.111258 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.111247 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7302a48-bc54-494e-bdea-b6f958088c7b","Type":"ContainerStarted","Data":"6b95af9b713d97ddc6d1d1c6472b128f74187f042f1fa314b108a94c8049fd0f"} Jun 08 11:25:27.135998 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.135952 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.1193712319999998 podStartE2EDuration="2.135937792s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.56656ms" isStatefulPod=false podCreationTimestamp="2026-06-08 11:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:25:27.134635951 +0000 UTC m=+316.619034821" watchObservedRunningTime="2026-06-08 11:25:27.135937792 +0000 UTC m=+316.620336654" Jun 08 11:25:27.254115 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.254082 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-58c8648587-jzgdp"] Jun 08 11:25:27.257650 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.257628 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.261656 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.260030 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jun 08 11:25:27.261656 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.260256 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jun 08 11:25:27.261656 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.260437 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jun 08 11:25:27.261656 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.261225 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wbrdf\"" Jun 08 11:25:27.261656 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.261416 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jun 08 11:25:27.261656 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.261645 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jun 08 11:25:27.265040 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.265016 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jun 08 11:25:27.270787 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.270713 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58c8648587-jzgdp"] Jun 08 11:25:27.445076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.445022 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-serving-certs-ca-bundle\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.445257 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.445117 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-metrics-client-ca\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.445257 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.445165 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-secret-telemeter-client\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.445257 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.445182 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-telemeter-client-tls\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.445257 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.445210 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lnpv\" (UniqueName: \"kubernetes.io/projected/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-kube-api-access-2lnpv\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.445257 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.445252 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.445427 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.445279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-federate-client-tls\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.445427 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.445372 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.546346 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.546254 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-metrics-client-ca\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.546346 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.546312 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-secret-telemeter-client\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.546346 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.546332 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-telemeter-client-tls\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.546593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.546436 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnpv\" (UniqueName: \"kubernetes.io/projected/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-kube-api-access-2lnpv\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.546593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.546471 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.546593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.546499 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-federate-client-tls\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.546593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.546523 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.546593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.546564 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-serving-certs-ca-bundle\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.547219 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.547186 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-metrics-client-ca\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.547514 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.547443 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-serving-certs-ca-bundle\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.547514 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.547491 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.549095 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.549073 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-secret-telemeter-client\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.549279 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.549257 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-telemeter-client-tls\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.549380 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.549258 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-federate-client-tls\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.550172 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.550148 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.555253 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.555221 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnpv\" (UniqueName: \"kubernetes.io/projected/79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6-kube-api-access-2lnpv\") pod \"telemeter-client-58c8648587-jzgdp\" (UID: \"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6\") " pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.569053 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.569028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" Jun 08 11:25:27.724689 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.724651 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58c8648587-jzgdp"] Jun 08 11:25:27.728932 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:25:27.728905 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f6dcb5_59cc_4b66_87c5_eb1cac2c86f6.slice/crio-395cdfbbec41080aadcadbae8ce8ccb671b70949ace9f67ffd96ba1b065f6e10 WatchSource:0}: Error finding container 395cdfbbec41080aadcadbae8ce8ccb671b70949ace9f67ffd96ba1b065f6e10: Status 404 returned error can't find the container with id 395cdfbbec41080aadcadbae8ce8ccb671b70949ace9f67ffd96ba1b065f6e10 Jun 08 11:25:27.731364 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:27.731335 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 08 11:25:28.116481 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:28.116448 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" event={"ID":"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6","Type":"ContainerStarted","Data":"395cdfbbec41080aadcadbae8ce8ccb671b70949ace9f67ffd96ba1b065f6e10"} Jun 08 11:25:30.125875 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:30.125839 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" event={"ID":"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6","Type":"ContainerStarted","Data":"43c27794be61fabb6246146d2e997c6d0caa82014a8363555076f018526b90d0"} Jun 08 11:25:30.126297 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:30.125893 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" event={"ID":"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6","Type":"ContainerStarted","Data":"bbbc00e60b3d279bca251b2eaffe36243beef88ca8a8a1726a940b096876e544"} Jun 08 11:25:30.126297 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:30.125913 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" event={"ID":"79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6","Type":"ContainerStarted","Data":"64eb59b1f8e049b43601143b83b23ef997c8f2bb16e4747d2165754eff74513a"} Jun 08 11:25:30.148696 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:25:30.148648 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-58c8648587-jzgdp" podStartSLOduration=1.260764019 podStartE2EDuration="3.148633266s" totalImagesPullingTime="1.887869247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:25:30.147336071 +0000 UTC m=+319.631734935" watchObservedRunningTime="2026-06-08 11:25:30.148633266 +0000 UTC m=+319.633032128" Jun 08 11:29:04.071319 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.071243 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss"] Jun 08 11:29:04.074623 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.074596 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" Jun 08 11:29:04.077124 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.077102 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-g4ccq\"" Jun 08 11:29:04.078107 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.078087 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jun 08 11:29:04.078201 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.078133 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jun 08 11:29:04.091487 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.091452 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pd7f\" (UniqueName: \"kubernetes.io/projected/60f9c948-6ae1-44ed-acf7-ed4f3d11e636-kube-api-access-9pd7f\") pod \"feast-operator-controller-manager-5f5976bd8-5rkss\" (UID: \"60f9c948-6ae1-44ed-acf7-ed4f3d11e636\") " pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" Jun 08 11:29:04.092096 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.092077 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss"] Jun 08 11:29:04.192479 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.192448 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pd7f\" (UniqueName: \"kubernetes.io/projected/60f9c948-6ae1-44ed-acf7-ed4f3d11e636-kube-api-access-9pd7f\") pod \"feast-operator-controller-manager-5f5976bd8-5rkss\" (UID: \"60f9c948-6ae1-44ed-acf7-ed4f3d11e636\") " pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" Jun 08 11:29:04.202045 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.202012 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pd7f\" (UniqueName: \"kubernetes.io/projected/60f9c948-6ae1-44ed-acf7-ed4f3d11e636-kube-api-access-9pd7f\") pod \"feast-operator-controller-manager-5f5976bd8-5rkss\" (UID: \"60f9c948-6ae1-44ed-acf7-ed4f3d11e636\") " pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" Jun 08 11:29:04.384553 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.384448 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" Jun 08 11:29:04.520030 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.519994 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss"] Jun 08 11:29:04.523229 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:29:04.523203 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f9c948_6ae1_44ed_acf7_ed4f3d11e636.slice/crio-0560e50da7596539d0e2de6b0a7fec322961480c435e126c6610fd4d82edb4f8 WatchSource:0}: Error finding container 0560e50da7596539d0e2de6b0a7fec322961480c435e126c6610fd4d82edb4f8: Status 404 returned error can't find the container with id 0560e50da7596539d0e2de6b0a7fec322961480c435e126c6610fd4d82edb4f8 Jun 08 11:29:04.982442 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:04.982406 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" event={"ID":"60f9c948-6ae1-44ed-acf7-ed4f3d11e636","Type":"ContainerStarted","Data":"0560e50da7596539d0e2de6b0a7fec322961480c435e126c6610fd4d82edb4f8"} Jun 08 11:29:07.997480 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:07.997443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" event={"ID":"60f9c948-6ae1-44ed-acf7-ed4f3d11e636","Type":"ContainerStarted","Data":"d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74"} Jun 08 11:29:07.997885 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:07.997524 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" Jun 08 11:29:08.013999 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:08.013941 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" podStartSLOduration=1.261394036 podStartE2EDuration="4.013925452s" totalImagesPullingTime="2.752531416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:29:08.012558851 +0000 UTC m=+537.496957714" watchObservedRunningTime="2026-06-08 11:29:08.013925452 +0000 UTC m=+537.498324315" Jun 08 11:29:19.003338 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:19.003295 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" Jun 08 11:29:47.246858 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.246826 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf"] Jun 08 11:29:47.250683 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.250664 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.253977 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.253949 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 08 11:29:47.254367 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.254339 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 08 11:29:47.254472 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.254347 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 08 11:29:47.254472 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.254386 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 08 11:29:47.254472 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.254429 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-k8ft5\"" Jun 08 11:29:47.254745 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.254729 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 08 11:29:47.254801 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.254746 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jun 08 11:29:47.254801 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.254774 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 08 11:29:47.265492 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.265464 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf"] Jun 08 11:29:47.372347 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.372310 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-online-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.372347 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.372346 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/38206ae1-e217-41ea-939f-f89e4eb80d18-feast-data\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.372644 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.372365 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf524\" (UniqueName: \"kubernetes.io/projected/38206ae1-e217-41ea-939f-f89e4eb80d18-kube-api-access-tf524\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.372644 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.372394 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-registry-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.372644 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.372444 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-offline-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.372644 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.372516 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-ui-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.473483 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.473441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-registry-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.473483 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.473484 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-offline-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.473846 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.473525 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-ui-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.473846 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.473576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-online-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.473846 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.473596 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/38206ae1-e217-41ea-939f-f89e4eb80d18-feast-data\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.473846 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:29:47.473638 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 08 11:29:47.473846 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.473650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf524\" (UniqueName: \"kubernetes.io/projected/38206ae1-e217-41ea-939f-f89e4eb80d18-kube-api-access-tf524\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.473846 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:29:47.473681 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 08 11:29:47.473846 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:29:47.473700 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jun 08 11:29:47.473846 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:29:47.473724 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-registry-tls podName:38206ae1-e217-41ea-939f-f89e4eb80d18 nodeName:}" failed. No retries permitted until 2026-06-08 11:29:47.973699385 +0000 UTC m=+577.458098226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-registry-tls") pod "feast-simple-feast-setup-55675d75f8-h5hmf" (UID: "38206ae1-e217-41ea-939f-f89e4eb80d18") : secret "feast-simple-feast-setup-registry-tls" not found Jun 08 11:29:47.473846 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:29:47.473756 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-offline-tls podName:38206ae1-e217-41ea-939f-f89e4eb80d18 nodeName:}" failed. No retries permitted until 2026-06-08 11:29:47.973738282 +0000 UTC m=+577.458137126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-offline-tls") pod "feast-simple-feast-setup-55675d75f8-h5hmf" (UID: "38206ae1-e217-41ea-939f-f89e4eb80d18") : secret "feast-simple-feast-setup-offline-tls" not found Jun 08 11:29:47.473846 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:29:47.473776 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-online-tls podName:38206ae1-e217-41ea-939f-f89e4eb80d18 nodeName:}" failed. No retries permitted until 2026-06-08 11:29:47.973769399 +0000 UTC m=+577.458168242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-online-tls") pod "feast-simple-feast-setup-55675d75f8-h5hmf" (UID: "38206ae1-e217-41ea-939f-f89e4eb80d18") : secret "feast-simple-feast-setup-online-tls" not found Jun 08 11:29:47.474175 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.474041 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/38206ae1-e217-41ea-939f-f89e4eb80d18-feast-data\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.476242 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.476226 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-ui-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.482896 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.482872 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf524\" (UniqueName: \"kubernetes.io/projected/38206ae1-e217-41ea-939f-f89e4eb80d18-kube-api-access-tf524\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.978086 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.978046 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-online-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.978275 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.978101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-registry-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.978321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.978280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-offline-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.980590 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.980556 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-registry-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.980734 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.980679 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-offline-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:47.980821 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:47.980799 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-online-tls\") pod \"feast-simple-feast-setup-55675d75f8-h5hmf\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:48.171705 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:48.171664 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:29:48.317961 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:48.317901 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf"] Jun 08 11:29:48.321645 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:29:48.321594 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38206ae1_e217_41ea_939f_f89e4eb80d18.slice/crio-7a73eb99ce1d0362c37d9708db744f4ac9ab261e3919b1d14674471bfbe20833 WatchSource:0}: Error finding container 7a73eb99ce1d0362c37d9708db744f4ac9ab261e3919b1d14674471bfbe20833: Status 404 returned error can't find the container with id 7a73eb99ce1d0362c37d9708db744f4ac9ab261e3919b1d14674471bfbe20833 Jun 08 11:29:49.166939 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:29:49.166896 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerStarted","Data":"7a73eb99ce1d0362c37d9708db744f4ac9ab261e3919b1d14674471bfbe20833"} Jun 08 11:30:17.301081 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:17.301045 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerStarted","Data":"2d0eeb7ee143e6f9a44ea2a9ba6f7a521ae37ab775486d88a66fcbbae40f7065"} Jun 08 11:30:21.319579 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:21.319541 2569 generic.go:356] "Generic (PLEG): container finished" podID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerID="2d0eeb7ee143e6f9a44ea2a9ba6f7a521ae37ab775486d88a66fcbbae40f7065" exitCode=0 Jun 08 11:30:21.319973 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:21.319634 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerDied","Data":"2d0eeb7ee143e6f9a44ea2a9ba6f7a521ae37ab775486d88a66fcbbae40f7065"} Jun 08 11:30:22.326304 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:22.326267 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerStarted","Data":"ba5e2bfcaf4f4ec1cfcaf2b5037706246bbb5899fcd06b618c6ed9ba043b2595"} Jun 08 11:30:27.349008 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:27.348971 2569 generic.go:356] "Generic (PLEG): container finished" podID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerID="ba5e2bfcaf4f4ec1cfcaf2b5037706246bbb5899fcd06b618c6ed9ba043b2595" exitCode=0 Jun 08 11:30:27.349384 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:27.349041 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerDied","Data":"ba5e2bfcaf4f4ec1cfcaf2b5037706246bbb5899fcd06b618c6ed9ba043b2595"} Jun 08 11:30:27.358922 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:27.358900 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jun 08 11:30:28.361005 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:28.360961 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerStarted","Data":"f2308946304cff1d5578ee0c3fde2dd05ff872ab2e3a8d9fd47f0806d631af7f"} Jun 08 11:30:28.361506 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:28.361031 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerStarted","Data":"8107d070207515ddbcf84a4ad58fc476895c8a4e3210f00771cd7578922118ce"} Jun 08 11:30:28.361506 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:28.361057 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerStarted","Data":"1b4aea34dff79a9be0fa88cb7157fc3adfd1e139958a040714bd658a79054b62"} Jun 08 11:30:28.361506 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:28.361075 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerStarted","Data":"ce3988cbc329d2211590b811eac854ed2b6c7c76c6a676c848d8238cdd78e4f2"} Jun 08 11:30:28.387388 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:28.387326 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podStartSLOduration=13.242659821 podStartE2EDuration="41.387300971s" totalImagesPullingTime="28.14464115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:30:28.385768525 +0000 UTC m=+617.870167393" watchObservedRunningTime="2026-06-08 11:30:28.387300971 +0000 UTC m=+617.871699835" Jun 08 11:30:30.172747 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:30.172708 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:30.173241 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:30.172765 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:30.173241 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:30.172784 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:30.173241 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:30.172800 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:30.175061 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:30.175014 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jun 08 11:30:30.175533 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:30.175038 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused" Jun 08 11:30:30.175678 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:30.175074 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused" Jun 08 11:30:30.175678 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:30.175084 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jun 08 11:30:33.173146 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.173119 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:33.173511 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.173234 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:33.173511 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.173437 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:33.173845 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.173821 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:33.173933 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.173867 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:33.173933 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.173892 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:33.174042 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.173988 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:33.174115 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.174094 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:33.174313 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.174294 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:33.186182 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.186162 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:33.391628 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.391593 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:30:33.394658 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:30:33.394600 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:31:34.597473 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.597438 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf"] Jun 08 11:31:34.598068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.597697 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="registry" containerID="cri-o://ce3988cbc329d2211590b811eac854ed2b6c7c76c6a676c848d8238cdd78e4f2" gracePeriod=30 Jun 08 11:31:34.598068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.597758 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="offline" containerID="cri-o://8107d070207515ddbcf84a4ad58fc476895c8a4e3210f00771cd7578922118ce" gracePeriod=30 Jun 08 11:31:34.598068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.597763 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="ui" containerID="cri-o://f2308946304cff1d5578ee0c3fde2dd05ff872ab2e3a8d9fd47f0806d631af7f" gracePeriod=30 Jun 08 11:31:34.598068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.597771 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="online" containerID="cri-o://1b4aea34dff79a9be0fa88cb7157fc3adfd1e139958a040714bd658a79054b62" gracePeriod=30 Jun 08 11:31:34.830410 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.830367 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2"] Jun 08 11:31:34.834730 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.834713 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:34.838107 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.838081 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-tzjh7\"" Jun 08 11:31:34.851439 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.851359 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2"] Jun 08 11:31:34.934596 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.934551 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-registry-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:34.934817 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.934648 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5tv\" (UniqueName: \"kubernetes.io/projected/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-kube-api-access-lj5tv\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:34.934817 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.934723 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-ui-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:34.934817 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.934761 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-offline-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:34.934817 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.934783 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-online-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:34.934977 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:34.934868 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-feast-data\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.035355 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.035313 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-ui-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.035647 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:31:35.035452 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 08 11:31:35.035822 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.035670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-offline-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.035925 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.035846 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-online-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.035925 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:31:35.035861 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-ui-tls podName:e208c14f-3f03-4c3b-8d48-99a172bb5ba9 nodeName:}" failed. No retries permitted until 2026-06-08 11:31:35.535834642 +0000 UTC m=+685.020233496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-ui-tls") pod "feast-simple-feast-setup-55675d75f8-px8l2" (UID: "e208c14f-3f03-4c3b-8d48-99a172bb5ba9") : secret "feast-simple-feast-setup-ui-tls" not found Jun 08 11:31:35.035925 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:31:35.035723 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 08 11:31:35.036064 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:31:35.035927 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-offline-tls podName:e208c14f-3f03-4c3b-8d48-99a172bb5ba9 nodeName:}" failed. No retries permitted until 2026-06-08 11:31:35.53590928 +0000 UTC m=+685.020308121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-offline-tls") pod "feast-simple-feast-setup-55675d75f8-px8l2" (UID: "e208c14f-3f03-4c3b-8d48-99a172bb5ba9") : secret "feast-simple-feast-setup-offline-tls" not found Jun 08 11:31:35.036064 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.035990 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-feast-data\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.036064 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.036054 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-registry-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.036646 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.036089 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5tv\" (UniqueName: \"kubernetes.io/projected/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-kube-api-access-lj5tv\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.036646 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:31:35.036178 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 08 11:31:35.036646 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:31:35.036237 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-registry-tls podName:e208c14f-3f03-4c3b-8d48-99a172bb5ba9 nodeName:}" failed. No retries permitted until 2026-06-08 11:31:35.536207555 +0000 UTC m=+685.020606396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-registry-tls") pod "feast-simple-feast-setup-55675d75f8-px8l2" (UID: "e208c14f-3f03-4c3b-8d48-99a172bb5ba9") : secret "feast-simple-feast-setup-registry-tls" not found Jun 08 11:31:35.036646 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.036472 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-feast-data\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.038910 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.038890 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-online-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.047828 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.047798 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5tv\" (UniqueName: \"kubernetes.io/projected/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-kube-api-access-lj5tv\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.540829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.540785 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-registry-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.541024 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.540852 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-ui-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.541024 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.540889 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-offline-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.543513 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.543477 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-offline-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.543513 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.543486 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-registry-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.543513 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.543508 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-ui-tls\") pod \"feast-simple-feast-setup-55675d75f8-px8l2\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.652693 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.652650 2569 generic.go:356] "Generic (PLEG): container finished" podID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerID="f2308946304cff1d5578ee0c3fde2dd05ff872ab2e3a8d9fd47f0806d631af7f" exitCode=0 Jun 08 11:31:35.652693 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.652681 2569 generic.go:356] "Generic (PLEG): container finished" podID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerID="1b4aea34dff79a9be0fa88cb7157fc3adfd1e139958a040714bd658a79054b62" exitCode=0 Jun 08 11:31:35.653118 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.652736 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerDied","Data":"f2308946304cff1d5578ee0c3fde2dd05ff872ab2e3a8d9fd47f0806d631af7f"} Jun 08 11:31:35.653118 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.652778 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerDied","Data":"1b4aea34dff79a9be0fa88cb7157fc3adfd1e139958a040714bd658a79054b62"} Jun 08 11:31:35.744558 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.744520 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:35.893201 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:35.893136 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2"] Jun 08 11:31:35.895548 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:31:35.895521 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode208c14f_3f03_4c3b_8d48_99a172bb5ba9.slice/crio-0c31e3ecdff15759eced5550cc0752c4f8e760ed5ffec3540ebb3b055e728c29 WatchSource:0}: Error finding container 0c31e3ecdff15759eced5550cc0752c4f8e760ed5ffec3540ebb3b055e728c29: Status 404 returned error can't find the container with id 0c31e3ecdff15759eced5550cc0752c4f8e760ed5ffec3540ebb3b055e728c29 Jun 08 11:31:36.658587 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:36.658550 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerStarted","Data":"7c2a1a543682a0d92b094eab0272ba34ab6bdd6a33ba52db0cb13f6efb98eca4"} Jun 08 11:31:36.659071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:36.658605 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerStarted","Data":"0c31e3ecdff15759eced5550cc0752c4f8e760ed5ffec3540ebb3b055e728c29"} Jun 08 11:31:40.680211 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:40.680177 2569 generic.go:356] "Generic (PLEG): container finished" podID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerID="7c2a1a543682a0d92b094eab0272ba34ab6bdd6a33ba52db0cb13f6efb98eca4" exitCode=0 Jun 08 11:31:40.680751 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:40.680256 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerDied","Data":"7c2a1a543682a0d92b094eab0272ba34ab6bdd6a33ba52db0cb13f6efb98eca4"} Jun 08 11:31:41.687913 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:41.687864 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerStarted","Data":"216b6eeeb43c09c4b75744072db51b78c512ef4a8e906018ea72429b7da49eef"} Jun 08 11:31:43.174087 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:43.174014 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jun 08 11:31:43.391772 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:43.391725 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jun 08 11:31:46.710770 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:46.710735 2569 generic.go:356] "Generic (PLEG): container finished" podID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerID="216b6eeeb43c09c4b75744072db51b78c512ef4a8e906018ea72429b7da49eef" exitCode=0 Jun 08 11:31:46.711164 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:46.710812 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerDied","Data":"216b6eeeb43c09c4b75744072db51b78c512ef4a8e906018ea72429b7da49eef"} Jun 08 11:31:47.722745 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.722706 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerStarted","Data":"5613b6c96dada1817ad274d8a46d503f0efc1ee8b31c8e1dcbae961934e3250d"} Jun 08 11:31:47.723239 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.722765 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerStarted","Data":"953da478313ab19bfbcfd7e96f794d227113e4404091c9b8dc85e56fa317d9f4"} Jun 08 11:31:47.723239 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.722785 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerStarted","Data":"2a9a9ab05481487714179670229c08cd2ab024a6d97b4de1d96d1497991b1d0d"} Jun 08 11:31:47.723239 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.722802 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerStarted","Data":"1ffbcdec0278a599810b0028c83cc27ff2c49e97adae28a3e7f4d7930e77494c"} Jun 08 11:31:47.744651 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.744584 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:47.744847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.744664 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:47.744847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.744685 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:47.744979 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.744909 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:47.748420 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.747848 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.26:6571: connect: connection refused" Jun 08 11:31:47.748420 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.747888 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Jun 08 11:31:47.748420 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.748041 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Jun 08 11:31:47.748420 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.748152 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.26:8816: connect: connection refused" Jun 08 11:31:47.748420 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:47.748277 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podStartSLOduration=13.748260006 podStartE2EDuration="13.748260006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:31:47.745917863 +0000 UTC m=+697.230316726" watchObservedRunningTime="2026-06-08 11:31:47.748260006 +0000 UTC m=+697.232658870" Jun 08 11:31:50.745094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:50.745048 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Jun 08 11:31:50.745680 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:50.745064 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.26:6571: connect: connection refused" Jun 08 11:31:50.745825 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:50.745065 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Jun 08 11:31:50.745825 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:50.745110 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.26:8816: connect: connection refused" Jun 08 11:31:53.174090 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.174039 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jun 08 11:31:53.392736 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.392645 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jun 08 11:31:53.745792 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.745760 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:53.745974 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.745874 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:53.746110 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.746091 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:53.746637 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.746420 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:53.746637 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.746520 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:53.746637 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.746541 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:53.746857 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.746692 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:53.746857 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.746758 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:53.746857 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.746830 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:53.750292 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.750275 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:53.771285 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.771258 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:31:53.774311 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:31:53.774288 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:32:03.174304 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:03.174258 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jun 08 11:32:03.174766 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:03.174405 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:32:03.391945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:03.391901 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jun 08 11:32:03.392126 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:03.392018 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:32:04.821072 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:04.821038 2569 generic.go:356] "Generic (PLEG): container finished" podID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerID="8107d070207515ddbcf84a4ad58fc476895c8a4e3210f00771cd7578922118ce" exitCode=137 Jun 08 11:32:04.821072 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:04.821065 2569 generic.go:356] "Generic (PLEG): container finished" podID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerID="ce3988cbc329d2211590b811eac854ed2b6c7c76c6a676c848d8238cdd78e4f2" exitCode=137 Jun 08 11:32:04.821479 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:04.821112 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerDied","Data":"8107d070207515ddbcf84a4ad58fc476895c8a4e3210f00771cd7578922118ce"} Jun 08 11:32:04.821479 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:04.821151 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerDied","Data":"ce3988cbc329d2211590b811eac854ed2b6c7c76c6a676c848d8238cdd78e4f2"} Jun 08 11:32:05.259768 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.259744 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:32:05.333905 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.333868 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-online-tls\") pod \"38206ae1-e217-41ea-939f-f89e4eb80d18\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " Jun 08 11:32:05.334086 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.333929 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-offline-tls\") pod \"38206ae1-e217-41ea-939f-f89e4eb80d18\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " Jun 08 11:32:05.334086 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.333960 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/38206ae1-e217-41ea-939f-f89e4eb80d18-feast-data\") pod \"38206ae1-e217-41ea-939f-f89e4eb80d18\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " Jun 08 11:32:05.334086 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.334012 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-registry-tls\") pod \"38206ae1-e217-41ea-939f-f89e4eb80d18\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " Jun 08 11:32:05.334086 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.334058 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-ui-tls\") pod \"38206ae1-e217-41ea-939f-f89e4eb80d18\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " Jun 08 11:32:05.334240 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.334099 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf524\" (UniqueName: \"kubernetes.io/projected/38206ae1-e217-41ea-939f-f89e4eb80d18-kube-api-access-tf524\") pod \"38206ae1-e217-41ea-939f-f89e4eb80d18\" (UID: \"38206ae1-e217-41ea-939f-f89e4eb80d18\") " Jun 08 11:32:05.334589 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.334525 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38206ae1-e217-41ea-939f-f89e4eb80d18-feast-data" (OuterVolumeSpecName: "feast-data") pod "38206ae1-e217-41ea-939f-f89e4eb80d18" (UID: "38206ae1-e217-41ea-939f-f89e4eb80d18"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:32:05.336458 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.336423 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-online-tls" (OuterVolumeSpecName: "online-tls") pod "38206ae1-e217-41ea-939f-f89e4eb80d18" (UID: "38206ae1-e217-41ea-939f-f89e4eb80d18"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:32:05.336458 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.336432 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38206ae1-e217-41ea-939f-f89e4eb80d18-kube-api-access-tf524" (OuterVolumeSpecName: "kube-api-access-tf524") pod "38206ae1-e217-41ea-939f-f89e4eb80d18" (UID: "38206ae1-e217-41ea-939f-f89e4eb80d18"). InnerVolumeSpecName "kube-api-access-tf524". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:32:05.336711 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.336490 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "38206ae1-e217-41ea-939f-f89e4eb80d18" (UID: "38206ae1-e217-41ea-939f-f89e4eb80d18"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:32:05.336770 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.336735 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "38206ae1-e217-41ea-939f-f89e4eb80d18" (UID: "38206ae1-e217-41ea-939f-f89e4eb80d18"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:32:05.336820 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.336766 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "38206ae1-e217-41ea-939f-f89e4eb80d18" (UID: "38206ae1-e217-41ea-939f-f89e4eb80d18"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:32:05.434935 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.434854 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-offline-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:32:05.434935 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.434880 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/38206ae1-e217-41ea-939f-f89e4eb80d18-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:32:05.434935 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.434893 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:32:05.434935 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.434906 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-ui-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:32:05.434935 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.434916 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf524\" (UniqueName: \"kubernetes.io/projected/38206ae1-e217-41ea-939f-f89e4eb80d18-kube-api-access-tf524\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:32:05.434935 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.434942 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/38206ae1-e217-41ea-939f-f89e4eb80d18-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:32:05.827994 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.827957 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" event={"ID":"38206ae1-e217-41ea-939f-f89e4eb80d18","Type":"ContainerDied","Data":"7a73eb99ce1d0362c37d9708db744f4ac9ab261e3919b1d14674471bfbe20833"} Jun 08 11:32:05.827994 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.827978 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf" Jun 08 11:32:05.828533 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.828020 2569 scope.go:117] "RemoveContainer" containerID="f2308946304cff1d5578ee0c3fde2dd05ff872ab2e3a8d9fd47f0806d631af7f" Jun 08 11:32:05.837983 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.837964 2569 scope.go:117] "RemoveContainer" containerID="8107d070207515ddbcf84a4ad58fc476895c8a4e3210f00771cd7578922118ce" Jun 08 11:32:05.845959 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.845934 2569 scope.go:117] "RemoveContainer" containerID="1b4aea34dff79a9be0fa88cb7157fc3adfd1e139958a040714bd658a79054b62" Jun 08 11:32:05.850203 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.850181 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf"] Jun 08 11:32:05.853790 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.853768 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-h5hmf"] Jun 08 11:32:05.854454 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.854438 2569 scope.go:117] "RemoveContainer" containerID="ce3988cbc329d2211590b811eac854ed2b6c7c76c6a676c848d8238cdd78e4f2" Jun 08 11:32:05.862400 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.862375 2569 scope.go:117] "RemoveContainer" containerID="ba5e2bfcaf4f4ec1cfcaf2b5037706246bbb5899fcd06b618c6ed9ba043b2595" Jun 08 11:32:05.869718 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:05.869702 2569 scope.go:117] "RemoveContainer" containerID="2d0eeb7ee143e6f9a44ea2a9ba6f7a521ae37ab775486d88a66fcbbae40f7065" Jun 08 11:32:06.953173 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:06.953132 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" path="/var/lib/kubelet/pods/38206ae1-e217-41ea-939f-f89e4eb80d18/volumes" Jun 08 11:32:56.470634 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.470578 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb"] Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.470973 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="feast-apply" Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.470988 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="feast-apply" Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.470998 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="registry" Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471003 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="registry" Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471010 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="ui" Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471016 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="ui" Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471028 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="feast-init" Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471034 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="feast-init" Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471042 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="offline" Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471047 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="offline" Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471057 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="online" Jun 08 11:32:56.471076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471062 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="online" Jun 08 11:32:56.471422 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471117 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="registry" Jun 08 11:32:56.471422 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471126 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="offline" Jun 08 11:32:56.471422 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471133 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="online" Jun 08 11:32:56.471422 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.471141 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="38206ae1-e217-41ea-939f-f89e4eb80d18" containerName="ui" Jun 08 11:32:56.474522 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.474503 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.477032 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.477005 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 08 11:32:56.477775 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.477757 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 08 11:32:56.477887 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.477791 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 08 11:32:56.477887 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.477826 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jun 08 11:32:56.477887 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.477866 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-pw5nc\"" Jun 08 11:32:56.478137 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.478119 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 08 11:32:56.478201 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.478126 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 08 11:32:56.478201 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.478154 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 08 11:32:56.480910 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.480885 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d90905f8-ba4a-4942-8b41-c0f23c18dd94-registry-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.481003 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.480953 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-offline-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.481003 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.480991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d90905f8-ba4a-4942-8b41-c0f23c18dd94-feast-data\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.481110 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.481019 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pfwd\" (UniqueName: \"kubernetes.io/projected/d90905f8-ba4a-4942-8b41-c0f23c18dd94-kube-api-access-4pfwd\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.481110 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.481101 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-online-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.481212 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.481179 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-ui-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.492500 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.492475 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb"] Jun 08 11:32:56.581893 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.581851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-offline-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.581893 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.581896 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d90905f8-ba4a-4942-8b41-c0f23c18dd94-feast-data\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.582173 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.581918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pfwd\" (UniqueName: \"kubernetes.io/projected/d90905f8-ba4a-4942-8b41-c0f23c18dd94-kube-api-access-4pfwd\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.582173 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.581949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-online-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.582173 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.581993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-ui-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.582173 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.582017 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d90905f8-ba4a-4942-8b41-c0f23c18dd94-registry-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.582173 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:32:56.582019 2569 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 08 11:32:56.582173 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:32:56.582112 2569 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jun 08 11:32:56.582173 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:32:56.582123 2569 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jun 08 11:32:56.582173 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:32:56.582128 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-offline-tls podName:d90905f8-ba4a-4942-8b41-c0f23c18dd94 nodeName:}" failed. No retries permitted until 2026-06-08 11:32:57.082106459 +0000 UTC m=+766.566505301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-offline-tls") pod "feast-simple-feast-remote-setup-654d458dd6-d42mb" (UID: "d90905f8-ba4a-4942-8b41-c0f23c18dd94") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 08 11:32:56.582485 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:32:56.582194 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-online-tls podName:d90905f8-ba4a-4942-8b41-c0f23c18dd94 nodeName:}" failed. No retries permitted until 2026-06-08 11:32:57.082176826 +0000 UTC m=+766.566575670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-online-tls") pod "feast-simple-feast-remote-setup-654d458dd6-d42mb" (UID: "d90905f8-ba4a-4942-8b41-c0f23c18dd94") : secret "feast-simple-feast-remote-setup-online-tls" not found Jun 08 11:32:56.582485 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:32:56.582210 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-ui-tls podName:d90905f8-ba4a-4942-8b41-c0f23c18dd94 nodeName:}" failed. No retries permitted until 2026-06-08 11:32:57.082201453 +0000 UTC m=+766.566600300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-ui-tls") pod "feast-simple-feast-remote-setup-654d458dd6-d42mb" (UID: "d90905f8-ba4a-4942-8b41-c0f23c18dd94") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jun 08 11:32:56.582485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.582291 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d90905f8-ba4a-4942-8b41-c0f23c18dd94-feast-data\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.582590 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.582581 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d90905f8-ba4a-4942-8b41-c0f23c18dd94-registry-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:56.592655 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:56.592626 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pfwd\" (UniqueName: \"kubernetes.io/projected/d90905f8-ba4a-4942-8b41-c0f23c18dd94-kube-api-access-4pfwd\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:57.087400 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:57.087362 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-ui-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:57.087585 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:57.087418 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-offline-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:57.087585 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:57.087455 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-online-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:57.090086 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:57.090047 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-online-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:57.090212 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:57.090130 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-ui-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:57.090212 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:57.090178 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-offline-tls\") pod \"feast-simple-feast-remote-setup-654d458dd6-d42mb\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:57.384202 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:57.384093 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:32:57.524122 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:57.524086 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb"] Jun 08 11:32:57.527271 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:32:57.527246 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd90905f8_ba4a_4942_8b41_c0f23c18dd94.slice/crio-250b76c1f8e1cdadd1d4f4eb77e2046f60b0eeb4e0f12ef4ff4404270e3a8922 WatchSource:0}: Error finding container 250b76c1f8e1cdadd1d4f4eb77e2046f60b0eeb4e0f12ef4ff4404270e3a8922: Status 404 returned error can't find the container with id 250b76c1f8e1cdadd1d4f4eb77e2046f60b0eeb4e0f12ef4ff4404270e3a8922 Jun 08 11:32:58.046330 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:58.046292 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerStarted","Data":"d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8"} Jun 08 11:32:58.046519 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:32:58.046345 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerStarted","Data":"250b76c1f8e1cdadd1d4f4eb77e2046f60b0eeb4e0f12ef4ff4404270e3a8922"} Jun 08 11:33:03.069986 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:03.069948 2569 generic.go:356] "Generic (PLEG): container finished" podID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerID="d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8" exitCode=0 Jun 08 11:33:03.070417 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:03.070022 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerDied","Data":"d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8"} Jun 08 11:33:04.076421 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:04.076387 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerStarted","Data":"7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e"} Jun 08 11:33:09.102166 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:09.102075 2569 generic.go:356] "Generic (PLEG): container finished" podID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerID="7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e" exitCode=0 Jun 08 11:33:09.102166 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:09.102148 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerDied","Data":"7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e"} Jun 08 11:33:10.113148 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:10.113112 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerStarted","Data":"84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5"} Jun 08 11:33:10.113513 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:10.113172 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerStarted","Data":"70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0"} Jun 08 11:33:10.113513 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:10.113194 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerStarted","Data":"1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9"} Jun 08 11:33:10.139423 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:10.139357 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podStartSLOduration=14.139332775 podStartE2EDuration="14.139332775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:33:10.137219018 +0000 UTC m=+779.621617913" watchObservedRunningTime="2026-06-08 11:33:10.139332775 +0000 UTC m=+779.623731639" Jun 08 11:33:12.384544 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:12.384492 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:33:12.384544 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:12.384554 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:33:12.385102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:12.384571 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:33:12.386506 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:12.386478 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Jun 08 11:33:12.386646 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:12.386512 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" Jun 08 11:33:12.386646 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:12.386581 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jun 08 11:33:15.385908 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:15.385879 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:33:15.386364 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:15.386310 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:33:15.386364 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:15.386344 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:33:15.386662 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:15.386643 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:33:15.390880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:15.390856 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:33:16.142927 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:16.142890 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:33:16.143104 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:16.142945 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:33:16.143637 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:16.143598 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:33:16.145891 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:33:16.145873 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:34:17.844595 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:17.844557 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb"] Jun 08 11:34:17.845374 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:17.845328 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="online" containerID="cri-o://1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9" gracePeriod=30 Jun 08 11:34:17.845573 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:17.845548 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="ui" containerID="cri-o://84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5" gracePeriod=30 Jun 08 11:34:17.845717 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:17.845696 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="offline" containerID="cri-o://70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0" gracePeriod=30 Jun 08 11:34:17.950438 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:17.950407 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2"] Jun 08 11:34:17.950847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:17.950704 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="online" containerID="cri-o://2a9a9ab05481487714179670229c08cd2ab024a6d97b4de1d96d1497991b1d0d" gracePeriod=30 Jun 08 11:34:17.950847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:17.950704 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="offline" containerID="cri-o://953da478313ab19bfbcfd7e96f794d227113e4404091c9b8dc85e56fa317d9f4" gracePeriod=30 Jun 08 11:34:17.950847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:17.950714 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="ui" containerID="cri-o://5613b6c96dada1817ad274d8a46d503f0efc1ee8b31c8e1dcbae961934e3250d" gracePeriod=30 Jun 08 11:34:17.950847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:17.950782 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="registry" containerID="cri-o://1ffbcdec0278a599810b0028c83cc27ff2c49e97adae28a3e7f4d7930e77494c" gracePeriod=30 Jun 08 11:34:19.420723 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:19.420679 2569 generic.go:356] "Generic (PLEG): container finished" podID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerID="5613b6c96dada1817ad274d8a46d503f0efc1ee8b31c8e1dcbae961934e3250d" exitCode=0 Jun 08 11:34:19.420723 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:19.420706 2569 generic.go:356] "Generic (PLEG): container finished" podID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerID="2a9a9ab05481487714179670229c08cd2ab024a6d97b4de1d96d1497991b1d0d" exitCode=0 Jun 08 11:34:19.421154 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:19.420757 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerDied","Data":"5613b6c96dada1817ad274d8a46d503f0efc1ee8b31c8e1dcbae961934e3250d"} Jun 08 11:34:19.421154 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:19.420801 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerDied","Data":"2a9a9ab05481487714179670229c08cd2ab024a6d97b4de1d96d1497991b1d0d"} Jun 08 11:34:19.423333 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:19.423307 2569 generic.go:356] "Generic (PLEG): container finished" podID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerID="84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5" exitCode=0 Jun 08 11:34:19.423333 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:19.423328 2569 generic.go:356] "Generic (PLEG): container finished" podID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerID="1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9" exitCode=0 Jun 08 11:34:19.423499 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:19.423375 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerDied","Data":"84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5"} Jun 08 11:34:19.423499 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:19.423405 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerDied","Data":"1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9"} Jun 08 11:34:23.747217 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:23.747169 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Jun 08 11:34:23.771976 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:23.771939 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Jun 08 11:34:26.143659 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:26.143594 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Jun 08 11:34:26.143659 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:26.143652 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jun 08 11:34:33.746677 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:33.746598 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Jun 08 11:34:33.772304 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:33.772262 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Jun 08 11:34:36.143319 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:36.143272 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Jun 08 11:34:36.143319 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:36.143303 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jun 08 11:34:43.746737 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:43.746696 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Jun 08 11:34:43.747141 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:43.746829 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:34:43.771500 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:43.771453 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Jun 08 11:34:43.771816 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:43.771795 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:34:46.143489 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:46.143445 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Jun 08 11:34:46.143895 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:46.143453 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jun 08 11:34:46.143895 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:46.143571 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:34:46.143895 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:46.143651 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:34:48.494782 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.494756 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:34:48.552233 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.552201 2569 generic.go:356] "Generic (PLEG): container finished" podID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerID="953da478313ab19bfbcfd7e96f794d227113e4404091c9b8dc85e56fa317d9f4" exitCode=137 Jun 08 11:34:48.552233 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.552228 2569 generic.go:356] "Generic (PLEG): container finished" podID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerID="1ffbcdec0278a599810b0028c83cc27ff2c49e97adae28a3e7f4d7930e77494c" exitCode=137 Jun 08 11:34:48.552439 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.552264 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerDied","Data":"953da478313ab19bfbcfd7e96f794d227113e4404091c9b8dc85e56fa317d9f4"} Jun 08 11:34:48.552439 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.552308 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerDied","Data":"1ffbcdec0278a599810b0028c83cc27ff2c49e97adae28a3e7f4d7930e77494c"} Jun 08 11:34:48.554674 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.554653 2569 generic.go:356] "Generic (PLEG): container finished" podID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerID="70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0" exitCode=137 Jun 08 11:34:48.554766 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.554715 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" Jun 08 11:34:48.554766 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.554734 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerDied","Data":"70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0"} Jun 08 11:34:48.554844 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.554777 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb" event={"ID":"d90905f8-ba4a-4942-8b41-c0f23c18dd94","Type":"ContainerDied","Data":"250b76c1f8e1cdadd1d4f4eb77e2046f60b0eeb4e0f12ef4ff4404270e3a8922"} Jun 08 11:34:48.554844 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.554802 2569 scope.go:117] "RemoveContainer" containerID="84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5" Jun 08 11:34:48.564083 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.564065 2569 scope.go:117] "RemoveContainer" containerID="70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0" Jun 08 11:34:48.572517 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.572489 2569 scope.go:117] "RemoveContainer" containerID="1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9" Jun 08 11:34:48.577689 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.577668 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d90905f8-ba4a-4942-8b41-c0f23c18dd94-registry-tls\") pod \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " Jun 08 11:34:48.577789 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.577695 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-offline-tls\") pod \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " Jun 08 11:34:48.577789 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.577736 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-online-tls\") pod \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " Jun 08 11:34:48.577906 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.577842 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pfwd\" (UniqueName: \"kubernetes.io/projected/d90905f8-ba4a-4942-8b41-c0f23c18dd94-kube-api-access-4pfwd\") pod \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " Jun 08 11:34:48.577962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.577911 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-ui-tls\") pod \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " Jun 08 11:34:48.577962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.577953 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d90905f8-ba4a-4942-8b41-c0f23c18dd94-feast-data\") pod \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\" (UID: \"d90905f8-ba4a-4942-8b41-c0f23c18dd94\") " Jun 08 11:34:48.578094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.578070 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90905f8-ba4a-4942-8b41-c0f23c18dd94-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d90905f8-ba4a-4942-8b41-c0f23c18dd94" (UID: "d90905f8-ba4a-4942-8b41-c0f23c18dd94"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:34:48.578550 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.578303 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d90905f8-ba4a-4942-8b41-c0f23c18dd94-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.578821 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.578793 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90905f8-ba4a-4942-8b41-c0f23c18dd94-feast-data" (OuterVolumeSpecName: "feast-data") pod "d90905f8-ba4a-4942-8b41-c0f23c18dd94" (UID: "d90905f8-ba4a-4942-8b41-c0f23c18dd94"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:34:48.581152 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.581121 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d90905f8-ba4a-4942-8b41-c0f23c18dd94" (UID: "d90905f8-ba4a-4942-8b41-c0f23c18dd94"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:34:48.581152 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.581143 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90905f8-ba4a-4942-8b41-c0f23c18dd94-kube-api-access-4pfwd" (OuterVolumeSpecName: "kube-api-access-4pfwd") pod "d90905f8-ba4a-4942-8b41-c0f23c18dd94" (UID: "d90905f8-ba4a-4942-8b41-c0f23c18dd94"). InnerVolumeSpecName "kube-api-access-4pfwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:34:48.581152 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.581143 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-online-tls" (OuterVolumeSpecName: "online-tls") pod "d90905f8-ba4a-4942-8b41-c0f23c18dd94" (UID: "d90905f8-ba4a-4942-8b41-c0f23c18dd94"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:34:48.581373 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.581178 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "d90905f8-ba4a-4942-8b41-c0f23c18dd94" (UID: "d90905f8-ba4a-4942-8b41-c0f23c18dd94"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:34:48.581898 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.581882 2569 scope.go:117] "RemoveContainer" containerID="7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e" Jun 08 11:34:48.591424 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.591403 2569 scope.go:117] "RemoveContainer" containerID="d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8" Jun 08 11:34:48.599865 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.599846 2569 scope.go:117] "RemoveContainer" containerID="84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5" Jun 08 11:34:48.600161 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:34:48.600131 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5\": container with ID starting with 84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5 not found: ID does not exist" containerID="84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5" Jun 08 11:34:48.600264 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.600175 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5"} err="failed to get container status \"84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5\": rpc error: code = NotFound desc = could not find container \"84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5\": container with ID starting with 84420488fae20c44db622064892e9151b07ed9f3ce67ee3447de0d1209aa9cc5 not found: ID does not exist" Jun 08 11:34:48.600264 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.600201 2569 scope.go:117] "RemoveContainer" containerID="70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0" Jun 08 11:34:48.600532 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:34:48.600471 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0\": container with ID starting with 70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0 not found: ID does not exist" containerID="70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0" Jun 08 11:34:48.600532 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.600493 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0"} err="failed to get container status \"70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0\": rpc error: code = NotFound desc = could not find container \"70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0\": container with ID starting with 70bdfec031db217dacaa6ee49948f18eb1d3e97613383df11932f5546f06c1b0 not found: ID does not exist" Jun 08 11:34:48.600532 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.600512 2569 scope.go:117] "RemoveContainer" containerID="1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9" Jun 08 11:34:48.600818 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:34:48.600800 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9\": container with ID starting with 1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9 not found: ID does not exist" containerID="1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9" Jun 08 11:34:48.600880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.600824 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9"} err="failed to get container status \"1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9\": rpc error: code = NotFound desc = could not find container \"1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9\": container with ID starting with 1436395cfd2ed6a78cfffd297b5310b939841bc6265d583c3b7b956751f41ad9 not found: ID does not exist" Jun 08 11:34:48.600880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.600848 2569 scope.go:117] "RemoveContainer" containerID="7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e" Jun 08 11:34:48.601072 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:34:48.601057 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e\": container with ID starting with 7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e not found: ID does not exist" containerID="7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e" Jun 08 11:34:48.601134 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.601080 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e"} err="failed to get container status \"7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e\": rpc error: code = NotFound desc = could not find container \"7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e\": container with ID starting with 7aa93bfc7ccd8c0a4fe1bb1ee513644d3bcba2ce435293c979ef2eef2ce8624e not found: ID does not exist" Jun 08 11:34:48.601134 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.601095 2569 scope.go:117] "RemoveContainer" containerID="d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8" Jun 08 11:34:48.601325 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:34:48.601310 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8\": container with ID starting with d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8 not found: ID does not exist" containerID="d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8" Jun 08 11:34:48.601370 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.601329 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8"} err="failed to get container status \"d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8\": rpc error: code = NotFound desc = could not find container \"d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8\": container with ID starting with d67862b578af3276c3e8210ad9bea396fffb93f1b589afe5c811d74079c3e0a8 not found: ID does not exist" Jun 08 11:34:48.611559 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.611539 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:34:48.679292 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679211 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-ui-tls\") pod \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " Jun 08 11:34:48.679292 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679245 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-registry-tls\") pod \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " Jun 08 11:34:48.679292 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679274 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-feast-data\") pod \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " Jun 08 11:34:48.679685 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679343 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-online-tls\") pod \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " Jun 08 11:34:48.679685 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679374 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-offline-tls\") pod \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " Jun 08 11:34:48.679685 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679416 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj5tv\" (UniqueName: \"kubernetes.io/projected/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-kube-api-access-lj5tv\") pod \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\" (UID: \"e208c14f-3f03-4c3b-8d48-99a172bb5ba9\") " Jun 08 11:34:48.679853 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679714 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-ui-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.679853 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679740 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d90905f8-ba4a-4942-8b41-c0f23c18dd94-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.679853 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679753 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-offline-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.679853 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679767 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d90905f8-ba4a-4942-8b41-c0f23c18dd94-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.679853 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679780 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pfwd\" (UniqueName: \"kubernetes.io/projected/d90905f8-ba4a-4942-8b41-c0f23c18dd94-kube-api-access-4pfwd\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.680267 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.679873 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-feast-data" (OuterVolumeSpecName: "feast-data") pod "e208c14f-3f03-4c3b-8d48-99a172bb5ba9" (UID: "e208c14f-3f03-4c3b-8d48-99a172bb5ba9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:34:48.681600 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.681572 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e208c14f-3f03-4c3b-8d48-99a172bb5ba9" (UID: "e208c14f-3f03-4c3b-8d48-99a172bb5ba9"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:34:48.682033 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.682013 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e208c14f-3f03-4c3b-8d48-99a172bb5ba9" (UID: "e208c14f-3f03-4c3b-8d48-99a172bb5ba9"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:34:48.682033 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.682024 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-online-tls" (OuterVolumeSpecName: "online-tls") pod "e208c14f-3f03-4c3b-8d48-99a172bb5ba9" (UID: "e208c14f-3f03-4c3b-8d48-99a172bb5ba9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:34:48.682120 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.682086 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e208c14f-3f03-4c3b-8d48-99a172bb5ba9" (UID: "e208c14f-3f03-4c3b-8d48-99a172bb5ba9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:34:48.682158 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.682116 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-kube-api-access-lj5tv" (OuterVolumeSpecName: "kube-api-access-lj5tv") pod "e208c14f-3f03-4c3b-8d48-99a172bb5ba9" (UID: "e208c14f-3f03-4c3b-8d48-99a172bb5ba9"). InnerVolumeSpecName "kube-api-access-lj5tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:34:48.780943 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.780903 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.780943 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.780937 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-offline-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.780943 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.780946 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj5tv\" (UniqueName: \"kubernetes.io/projected/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-kube-api-access-lj5tv\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.781180 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.780956 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-ui-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.781180 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.780966 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.781180 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.780975 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e208c14f-3f03-4c3b-8d48-99a172bb5ba9-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:34:48.886942 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.886912 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb"] Jun 08 11:34:48.891419 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.891392 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-654d458dd6-d42mb"] Jun 08 11:34:48.953702 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:48.953629 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" path="/var/lib/kubelet/pods/d90905f8-ba4a-4942-8b41-c0f23c18dd94/volumes" Jun 08 11:34:49.561736 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:49.561695 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" event={"ID":"e208c14f-3f03-4c3b-8d48-99a172bb5ba9","Type":"ContainerDied","Data":"0c31e3ecdff15759eced5550cc0752c4f8e760ed5ffec3540ebb3b055e728c29"} Jun 08 11:34:49.561736 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:49.561732 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2" Jun 08 11:34:49.562260 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:49.561755 2569 scope.go:117] "RemoveContainer" containerID="5613b6c96dada1817ad274d8a46d503f0efc1ee8b31c8e1dcbae961934e3250d" Jun 08 11:34:49.570743 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:49.570723 2569 scope.go:117] "RemoveContainer" containerID="953da478313ab19bfbcfd7e96f794d227113e4404091c9b8dc85e56fa317d9f4" Jun 08 11:34:49.578530 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:49.578509 2569 scope.go:117] "RemoveContainer" containerID="2a9a9ab05481487714179670229c08cd2ab024a6d97b4de1d96d1497991b1d0d" Jun 08 11:34:49.586515 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:49.586490 2569 scope.go:117] "RemoveContainer" containerID="1ffbcdec0278a599810b0028c83cc27ff2c49e97adae28a3e7f4d7930e77494c" Jun 08 11:34:49.586829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:49.586808 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2"] Jun 08 11:34:49.590789 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:49.590762 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-55675d75f8-px8l2"] Jun 08 11:34:49.594820 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:49.594802 2569 scope.go:117] "RemoveContainer" containerID="216b6eeeb43c09c4b75744072db51b78c512ef4a8e906018ea72429b7da49eef" Jun 08 11:34:49.602214 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:49.602194 2569 scope.go:117] "RemoveContainer" containerID="7c2a1a543682a0d92b094eab0272ba34ab6bdd6a33ba52db0cb13f6efb98eca4" Jun 08 11:34:50.953842 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:34:50.953814 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" path="/var/lib/kubelet/pods/e208c14f-3f03-4c3b-8d48-99a172bb5ba9/volumes" Jun 08 11:35:00.802819 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.802731 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-phqhl"] Jun 08 11:35:00.803263 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803207 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="offline" Jun 08 11:35:00.803263 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803226 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="offline" Jun 08 11:35:00.803263 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803237 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="feast-init" Jun 08 11:35:00.803263 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803246 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="feast-init" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803269 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="ui" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803278 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="ui" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803288 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="online" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803296 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="online" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803313 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="feast-apply" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803322 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="feast-apply" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803332 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="registry" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803340 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="registry" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803352 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="offline" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803360 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="offline" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803373 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="ui" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803381 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="ui" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803408 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="feast-init" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803417 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="feast-init" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803425 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="online" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803433 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="online" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803442 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="feast-apply" Jun 08 11:35:00.803467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803450 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="feast-apply" Jun 08 11:35:00.804308 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803528 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="ui" Jun 08 11:35:00.804308 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803542 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="ui" Jun 08 11:35:00.804308 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803551 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="online" Jun 08 11:35:00.804308 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803561 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="offline" Jun 08 11:35:00.804308 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803571 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="offline" Jun 08 11:35:00.804308 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803579 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e208c14f-3f03-4c3b-8d48-99a172bb5ba9" containerName="registry" Jun 08 11:35:00.804308 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.803587 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90905f8-ba4a-4942-8b41-c0f23c18dd94" containerName="online" Jun 08 11:35:00.806969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.806948 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-phqhl" Jun 08 11:35:00.809477 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.809452 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 08 11:35:00.809680 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.809458 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 08 11:35:00.809680 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.809548 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-kzxwr\"" Jun 08 11:35:00.812201 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.812176 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-phqhl"] Jun 08 11:35:00.881677 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.881645 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vn8s\" (UniqueName: \"kubernetes.io/projected/b838179f-5a56-4662-bf1e-b66954627c9b-kube-api-access-2vn8s\") pod \"redis-69bbd798b6-phqhl\" (UID: \"b838179f-5a56-4662-bf1e-b66954627c9b\") " pod="test-ns-feast/redis-69bbd798b6-phqhl" Jun 08 11:35:00.982997 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.982945 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vn8s\" (UniqueName: \"kubernetes.io/projected/b838179f-5a56-4662-bf1e-b66954627c9b-kube-api-access-2vn8s\") pod \"redis-69bbd798b6-phqhl\" (UID: \"b838179f-5a56-4662-bf1e-b66954627c9b\") " pod="test-ns-feast/redis-69bbd798b6-phqhl" Jun 08 11:35:00.991383 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:00.991356 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vn8s\" (UniqueName: \"kubernetes.io/projected/b838179f-5a56-4662-bf1e-b66954627c9b-kube-api-access-2vn8s\") pod \"redis-69bbd798b6-phqhl\" (UID: \"b838179f-5a56-4662-bf1e-b66954627c9b\") " pod="test-ns-feast/redis-69bbd798b6-phqhl" Jun 08 11:35:01.125846 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:01.125766 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-phqhl" Jun 08 11:35:01.260756 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:01.260716 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-phqhl"] Jun 08 11:35:01.264861 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:35:01.264832 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb838179f_5a56_4662_bf1e_b66954627c9b.slice/crio-c9d3df45ce05055275fecb14071cc7ba3bf93a0aec14eb7f46e7391a24600b1e WatchSource:0}: Error finding container c9d3df45ce05055275fecb14071cc7ba3bf93a0aec14eb7f46e7391a24600b1e: Status 404 returned error can't find the container with id c9d3df45ce05055275fecb14071cc7ba3bf93a0aec14eb7f46e7391a24600b1e Jun 08 11:35:01.266655 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:01.266637 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 08 11:35:01.614316 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:01.614277 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-phqhl" event={"ID":"b838179f-5a56-4662-bf1e-b66954627c9b","Type":"ContainerStarted","Data":"c9d3df45ce05055275fecb14071cc7ba3bf93a0aec14eb7f46e7391a24600b1e"} Jun 08 11:35:03.624456 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:03.624417 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-phqhl" event={"ID":"b838179f-5a56-4662-bf1e-b66954627c9b","Type":"ContainerStarted","Data":"87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf"} Jun 08 11:35:03.640094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:03.640041 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-phqhl" podStartSLOduration=2.21438577 podStartE2EDuration="3.640024563s" totalImagesPullingTime="1.425638793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:35:03.638860493 +0000 UTC m=+893.123259356" watchObservedRunningTime="2026-06-08 11:35:03.640024563 +0000 UTC m=+893.124423426" Jun 08 11:35:11.437330 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.437255 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf"] Jun 08 11:35:11.442298 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.442280 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.444768 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.444738 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jun 08 11:35:11.444768 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.444751 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-4fl28\"" Jun 08 11:35:11.444948 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.444807 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jun 08 11:35:11.444948 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.444754 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jun 08 11:35:11.444948 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.444881 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jun 08 11:35:11.445071 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.445032 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jun 08 11:35:11.451079 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.451055 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf"] Jun 08 11:35:11.574341 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.574299 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-offline-tls\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.574544 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.574356 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-registry-tls\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.574544 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.574389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c125d9-0b4e-47cf-a106-3feb4765bb39-feast-data\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.574544 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.574431 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-online-tls\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.574544 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.574520 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4znk\" (UniqueName: \"kubernetes.io/projected/c1c125d9-0b4e-47cf-a106-3feb4765bb39-kube-api-access-p4znk\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.675028 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.674994 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-online-tls\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.675028 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.675042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4znk\" (UniqueName: \"kubernetes.io/projected/c1c125d9-0b4e-47cf-a106-3feb4765bb39-kube-api-access-p4znk\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.675291 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.675076 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-offline-tls\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.675291 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.675112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-registry-tls\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.675291 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.675135 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c125d9-0b4e-47cf-a106-3feb4765bb39-feast-data\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.675291 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:35:11.675222 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Jun 08 11:35:11.675493 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:35:11.675297 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-offline-tls podName:c1c125d9-0b4e-47cf-a106-3feb4765bb39 nodeName:}" failed. No retries permitted until 2026-06-08 11:35:12.175276968 +0000 UTC m=+901.659675808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-offline-tls") pod "feast-credit-scoring-bf686bf5f-kl2gf" (UID: "c1c125d9-0b4e-47cf-a106-3feb4765bb39") : secret "feast-credit-scoring-offline-tls" not found Jun 08 11:35:11.675579 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.675558 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c125d9-0b4e-47cf-a106-3feb4765bb39-feast-data\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.677727 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.677707 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-online-tls\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.677727 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.677726 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-registry-tls\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:11.683841 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:11.683810 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4znk\" (UniqueName: \"kubernetes.io/projected/c1c125d9-0b4e-47cf-a106-3feb4765bb39-kube-api-access-p4znk\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:12.180036 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:12.179996 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-offline-tls\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:12.182553 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:12.182517 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-offline-tls\") pod \"feast-credit-scoring-bf686bf5f-kl2gf\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:12.352811 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:12.352763 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:12.493020 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:12.492978 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf"] Jun 08 11:35:12.500992 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:35:12.500949 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c125d9_0b4e_47cf_a106_3feb4765bb39.slice/crio-114b309c36c3b4f8a45c64448112813762861802c64e49ffa5aee262ea832d9f WatchSource:0}: Error finding container 114b309c36c3b4f8a45c64448112813762861802c64e49ffa5aee262ea832d9f: Status 404 returned error can't find the container with id 114b309c36c3b4f8a45c64448112813762861802c64e49ffa5aee262ea832d9f Jun 08 11:35:12.662136 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:12.662095 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerStarted","Data":"f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f"} Jun 08 11:35:12.662305 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:12.662155 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerStarted","Data":"114b309c36c3b4f8a45c64448112813762861802c64e49ffa5aee262ea832d9f"} Jun 08 11:35:13.667737 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:13.667704 2569 generic.go:356] "Generic (PLEG): container finished" podID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerID="f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f" exitCode=0 Jun 08 11:35:13.668144 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:13.667750 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerDied","Data":"f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f"} Jun 08 11:35:14.673770 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:14.673736 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerStarted","Data":"110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda"} Jun 08 11:35:20.701504 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:20.701470 2569 generic.go:356] "Generic (PLEG): container finished" podID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerID="110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda" exitCode=0 Jun 08 11:35:20.701937 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:20.701530 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerDied","Data":"110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda"} Jun 08 11:35:21.710499 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:21.710445 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerStarted","Data":"fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca"} Jun 08 11:35:21.711012 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:21.710526 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerStarted","Data":"9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67"} Jun 08 11:35:21.711012 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:21.710557 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerStarted","Data":"8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0"} Jun 08 11:35:21.738252 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:21.738197 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podStartSLOduration=10.738181128 podStartE2EDuration="10.738181128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:35:21.736636407 +0000 UTC m=+911.221035266" watchObservedRunningTime="2026-06-08 11:35:21.738181128 +0000 UTC m=+911.222579991" Jun 08 11:35:22.353390 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:22.353346 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:22.353694 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:22.353438 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:22.353694 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:22.353470 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:24.353870 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:24.353823 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:24.354338 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:24.353906 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:24.354338 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:24.353930 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:24.355794 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:24.355744 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused" Jun 08 11:35:24.355949 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:24.355808 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jun 08 11:35:24.355949 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:24.355770 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused" Jun 08 11:35:27.353966 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:27.353916 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused" Jun 08 11:35:27.354901 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:27.354878 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:27.355754 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:27.355734 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:27.361412 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:27.361394 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:27.747083 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:27.747052 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:30.353522 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:30.353478 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused" Jun 08 11:35:33.353983 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:33.353946 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:35:33.354719 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:35:33.354701 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:36:37.138548 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:37.138502 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf"] Jun 08 11:36:37.139052 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:37.138785 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="registry" containerID="cri-o://8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0" gracePeriod=30 Jun 08 11:36:37.139052 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:37.138845 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="offline" containerID="cri-o://fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca" gracePeriod=30 Jun 08 11:36:37.139052 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:37.138857 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="online" containerID="cri-o://9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67" gracePeriod=30 Jun 08 11:36:37.160734 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:37.160700 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-phqhl"] Jun 08 11:36:37.160942 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:37.160911 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-phqhl" podUID="b838179f-5a56-4662-bf1e-b66954627c9b" containerName="redis" containerID="cri-o://87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf" gracePeriod=30 Jun 08 11:36:37.413431 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:37.413400 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-phqhl" Jun 08 11:36:37.544523 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:37.544491 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vn8s\" (UniqueName: \"kubernetes.io/projected/b838179f-5a56-4662-bf1e-b66954627c9b-kube-api-access-2vn8s\") pod \"b838179f-5a56-4662-bf1e-b66954627c9b\" (UID: \"b838179f-5a56-4662-bf1e-b66954627c9b\") " Jun 08 11:36:37.546717 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:37.546680 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b838179f-5a56-4662-bf1e-b66954627c9b-kube-api-access-2vn8s" (OuterVolumeSpecName: "kube-api-access-2vn8s") pod "b838179f-5a56-4662-bf1e-b66954627c9b" (UID: "b838179f-5a56-4662-bf1e-b66954627c9b"). InnerVolumeSpecName "kube-api-access-2vn8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:36:37.645364 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:37.645268 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vn8s\" (UniqueName: \"kubernetes.io/projected/b838179f-5a56-4662-bf1e-b66954627c9b-kube-api-access-2vn8s\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:36:37.743981 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:37.743923 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jun 08 11:36:38.058680 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.058644 2569 generic.go:356] "Generic (PLEG): container finished" podID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerID="9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67" exitCode=0 Jun 08 11:36:38.058849 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.058720 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerDied","Data":"9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67"} Jun 08 11:36:38.060010 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.059989 2569 generic.go:356] "Generic (PLEG): container finished" podID="b838179f-5a56-4662-bf1e-b66954627c9b" containerID="87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf" exitCode=0 Jun 08 11:36:38.060140 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.060033 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-phqhl" event={"ID":"b838179f-5a56-4662-bf1e-b66954627c9b","Type":"ContainerDied","Data":"87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf"} Jun 08 11:36:38.060140 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.060043 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-phqhl" Jun 08 11:36:38.060140 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.060058 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-phqhl" event={"ID":"b838179f-5a56-4662-bf1e-b66954627c9b","Type":"ContainerDied","Data":"c9d3df45ce05055275fecb14071cc7ba3bf93a0aec14eb7f46e7391a24600b1e"} Jun 08 11:36:38.060140 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.060074 2569 scope.go:117] "RemoveContainer" containerID="87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf" Jun 08 11:36:38.075735 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.075715 2569 scope.go:117] "RemoveContainer" containerID="87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf" Jun 08 11:36:38.076016 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:36:38.075995 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf\": container with ID starting with 87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf not found: ID does not exist" containerID="87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf" Jun 08 11:36:38.076075 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.076025 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf"} err="failed to get container status \"87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf\": rpc error: code = NotFound desc = could not find container \"87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf\": container with ID starting with 87085cef2e696698d3921f81fd3ff050842fa7810249cd4cb29df2e2c8e82cdf not found: ID does not exist" Jun 08 11:36:38.083170 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.083139 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-phqhl"] Jun 08 11:36:38.086293 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.086271 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-phqhl"] Jun 08 11:36:38.956169 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:38.956133 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b838179f-5a56-4662-bf1e-b66954627c9b" path="/var/lib/kubelet/pods/b838179f-5a56-4662-bf1e-b66954627c9b/volumes" Jun 08 11:36:47.743733 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:47.743681 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jun 08 11:36:57.743774 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:57.743731 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Jun 08 11:36:57.744255 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:36:57.743874 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:37:07.355515 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.355471 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused" Jun 08 11:37:07.785231 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.785208 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:37:07.795863 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.795836 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4znk\" (UniqueName: \"kubernetes.io/projected/c1c125d9-0b4e-47cf-a106-3feb4765bb39-kube-api-access-p4znk\") pod \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " Jun 08 11:37:07.795983 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.795897 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c125d9-0b4e-47cf-a106-3feb4765bb39-feast-data\") pod \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " Jun 08 11:37:07.795983 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.795933 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-offline-tls\") pod \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " Jun 08 11:37:07.795983 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.795950 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-online-tls\") pod \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " Jun 08 11:37:07.795983 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.795968 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-registry-tls\") pod \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\" (UID: \"c1c125d9-0b4e-47cf-a106-3feb4765bb39\") " Jun 08 11:37:07.798213 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.798177 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-online-tls" (OuterVolumeSpecName: "online-tls") pod "c1c125d9-0b4e-47cf-a106-3feb4765bb39" (UID: "c1c125d9-0b4e-47cf-a106-3feb4765bb39"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:37:07.798321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.798250 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c1c125d9-0b4e-47cf-a106-3feb4765bb39" (UID: "c1c125d9-0b4e-47cf-a106-3feb4765bb39"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:37:07.798321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.798262 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c125d9-0b4e-47cf-a106-3feb4765bb39-kube-api-access-p4znk" (OuterVolumeSpecName: "kube-api-access-p4znk") pod "c1c125d9-0b4e-47cf-a106-3feb4765bb39" (UID: "c1c125d9-0b4e-47cf-a106-3feb4765bb39"). InnerVolumeSpecName "kube-api-access-p4znk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:37:07.798463 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.798438 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c1c125d9-0b4e-47cf-a106-3feb4765bb39" (UID: "c1c125d9-0b4e-47cf-a106-3feb4765bb39"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:37:07.805891 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.805864 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c125d9-0b4e-47cf-a106-3feb4765bb39-feast-data" (OuterVolumeSpecName: "feast-data") pod "c1c125d9-0b4e-47cf-a106-3feb4765bb39" (UID: "c1c125d9-0b4e-47cf-a106-3feb4765bb39"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:37:07.896887 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.896849 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4znk\" (UniqueName: \"kubernetes.io/projected/c1c125d9-0b4e-47cf-a106-3feb4765bb39-kube-api-access-p4znk\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:37:07.896887 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.896880 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c125d9-0b4e-47cf-a106-3feb4765bb39-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:37:07.896887 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.896890 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-offline-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:37:07.896887 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.896898 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:37:07.897143 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:07.896907 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c1c125d9-0b4e-47cf-a106-3feb4765bb39-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:37:08.189464 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.189416 2569 generic.go:356] "Generic (PLEG): container finished" podID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerID="fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca" exitCode=137 Jun 08 11:37:08.189464 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.189456 2569 generic.go:356] "Generic (PLEG): container finished" podID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerID="8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0" exitCode=137 Jun 08 11:37:08.189692 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.189491 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" Jun 08 11:37:08.189692 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.189483 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerDied","Data":"fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca"} Jun 08 11:37:08.189692 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.189625 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerDied","Data":"8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0"} Jun 08 11:37:08.189692 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.189646 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" event={"ID":"c1c125d9-0b4e-47cf-a106-3feb4765bb39","Type":"ContainerDied","Data":"114b309c36c3b4f8a45c64448112813762861802c64e49ffa5aee262ea832d9f"} Jun 08 11:37:08.189692 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.189667 2569 scope.go:117] "RemoveContainer" containerID="fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca" Jun 08 11:37:08.199499 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.199478 2569 scope.go:117] "RemoveContainer" containerID="9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67" Jun 08 11:37:08.207478 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.207461 2569 scope.go:117] "RemoveContainer" containerID="8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0" Jun 08 11:37:08.210568 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.210545 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf"] Jun 08 11:37:08.214620 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.214581 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf"] Jun 08 11:37:08.218806 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.218784 2569 scope.go:117] "RemoveContainer" containerID="110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda" Jun 08 11:37:08.226515 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.226497 2569 scope.go:117] "RemoveContainer" containerID="f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f" Jun 08 11:37:08.233967 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.233946 2569 scope.go:117] "RemoveContainer" containerID="fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca" Jun 08 11:37:08.234236 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:37:08.234210 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca\": container with ID starting with fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca not found: ID does not exist" containerID="fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca" Jun 08 11:37:08.234277 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.234248 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca"} err="failed to get container status \"fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca\": rpc error: code = NotFound desc = could not find container \"fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca\": container with ID starting with fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca not found: ID does not exist" Jun 08 11:37:08.234277 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.234268 2569 scope.go:117] "RemoveContainer" containerID="9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67" Jun 08 11:37:08.234454 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:37:08.234433 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67\": container with ID starting with 9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67 not found: ID does not exist" containerID="9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67" Jun 08 11:37:08.234501 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.234461 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67"} err="failed to get container status \"9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67\": rpc error: code = NotFound desc = could not find container \"9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67\": container with ID starting with 9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67 not found: ID does not exist" Jun 08 11:37:08.234501 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.234477 2569 scope.go:117] "RemoveContainer" containerID="8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0" Jun 08 11:37:08.234705 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:37:08.234686 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0\": container with ID starting with 8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0 not found: ID does not exist" containerID="8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0" Jun 08 11:37:08.234753 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.234711 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0"} err="failed to get container status \"8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0\": rpc error: code = NotFound desc = could not find container \"8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0\": container with ID starting with 8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0 not found: ID does not exist" Jun 08 11:37:08.234753 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.234725 2569 scope.go:117] "RemoveContainer" containerID="110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda" Jun 08 11:37:08.234982 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:37:08.234959 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda\": container with ID starting with 110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda not found: ID does not exist" containerID="110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda" Jun 08 11:37:08.235020 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.234987 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda"} err="failed to get container status \"110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda\": rpc error: code = NotFound desc = could not find container \"110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda\": container with ID starting with 110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda not found: ID does not exist" Jun 08 11:37:08.235020 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.235002 2569 scope.go:117] "RemoveContainer" containerID="f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f" Jun 08 11:37:08.235234 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:37:08.235215 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f\": container with ID starting with f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f not found: ID does not exist" containerID="f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f" Jun 08 11:37:08.235309 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.235240 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f"} err="failed to get container status \"f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f\": rpc error: code = NotFound desc = could not find container \"f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f\": container with ID starting with f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f not found: ID does not exist" Jun 08 11:37:08.235309 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.235260 2569 scope.go:117] "RemoveContainer" containerID="fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca" Jun 08 11:37:08.235478 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.235461 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca"} err="failed to get container status \"fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca\": rpc error: code = NotFound desc = could not find container \"fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca\": container with ID starting with fe9238fbde7c91328032d6dd0e93b6f046ef8c59f32f485d2d52345550308aca not found: ID does not exist" Jun 08 11:37:08.235545 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.235481 2569 scope.go:117] "RemoveContainer" containerID="9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67" Jun 08 11:37:08.235736 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.235717 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67"} err="failed to get container status \"9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67\": rpc error: code = NotFound desc = could not find container \"9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67\": container with ID starting with 9f1ca3ddf40a91f4dfddb536e021dd67ca65b17842f9697cad9a0f342610aa67 not found: ID does not exist" Jun 08 11:37:08.235736 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.235736 2569 scope.go:117] "RemoveContainer" containerID="8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0" Jun 08 11:37:08.235962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.235942 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0"} err="failed to get container status \"8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0\": rpc error: code = NotFound desc = could not find container \"8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0\": container with ID starting with 8b77d3f30d269faef3ff6106ccc28634d077d66f12e97bc89958871eac99cbd0 not found: ID does not exist" Jun 08 11:37:08.236011 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.235964 2569 scope.go:117] "RemoveContainer" containerID="110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda" Jun 08 11:37:08.236180 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.236164 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda"} err="failed to get container status \"110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda\": rpc error: code = NotFound desc = could not find container \"110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda\": container with ID starting with 110e2aabaf33c4cc5ff3fb2b77762cbfa6216eca78ef0998fc2f175ae025ecda not found: ID does not exist" Jun 08 11:37:08.236220 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.236180 2569 scope.go:117] "RemoveContainer" containerID="f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f" Jun 08 11:37:08.236391 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.236373 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f"} err="failed to get container status \"f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f\": rpc error: code = NotFound desc = could not find container \"f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f\": container with ID starting with f91551390655c48ac3d9f66f0c223026c07b81c3de4f641b7949faff12de3e8f not found: ID does not exist" Jun 08 11:37:08.743810 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.743763 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bf686bf5f-kl2gf" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": context deadline exceeded" Jun 08 11:37:08.953872 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:37:08.953841 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" path="/var/lib/kubelet/pods/c1c125d9-0b4e-47cf-a106-3feb4765bb39/volumes" Jun 08 11:39:25.952359 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952272 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk"] Jun 08 11:39:25.952880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952752 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b838179f-5a56-4662-bf1e-b66954627c9b" containerName="redis" Jun 08 11:39:25.952880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952771 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b838179f-5a56-4662-bf1e-b66954627c9b" containerName="redis" Jun 08 11:39:25.952880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952783 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="feast-apply" Jun 08 11:39:25.952880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952791 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="feast-apply" Jun 08 11:39:25.952880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952811 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="feast-init" Jun 08 11:39:25.952880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952818 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="feast-init" Jun 08 11:39:25.952880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952830 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="offline" Jun 08 11:39:25.952880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952839 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="offline" Jun 08 11:39:25.952880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952856 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="registry" Jun 08 11:39:25.952880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952864 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="registry" Jun 08 11:39:25.952880 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952883 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="online" Jun 08 11:39:25.953408 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952891 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="online" Jun 08 11:39:25.953408 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.952996 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="online" Jun 08 11:39:25.953408 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.953015 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b838179f-5a56-4662-bf1e-b66954627c9b" containerName="redis" Jun 08 11:39:25.953408 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.953024 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="registry" Jun 08 11:39:25.953408 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.953033 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1c125d9-0b4e-47cf-a106-3feb4765bb39" containerName="offline" Jun 08 11:39:25.956297 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.956276 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:25.958552 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.958515 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jun 08 11:39:25.958789 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.958771 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-wwn4g\"" Jun 08 11:39:25.958860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.958798 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jun 08 11:39:25.958860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.958806 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jun 08 11:39:25.958860 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.958806 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jun 08 11:39:25.958980 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.958777 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jun 08 11:39:25.959494 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.959478 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jun 08 11:39:25.966222 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:25.966179 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk"] Jun 08 11:39:26.070079 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.070035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-registry-tls\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.070079 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.070080 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-online-tls\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.070331 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.070200 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bft6\" (UniqueName: \"kubernetes.io/projected/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-kube-api-access-2bft6\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.070331 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.070267 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-feast-data\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.170782 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.170747 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-registry-tls\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.170782 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.170781 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-online-tls\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.171012 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.170802 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bft6\" (UniqueName: \"kubernetes.io/projected/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-kube-api-access-2bft6\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.171012 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.170838 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-feast-data\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.171168 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.171150 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-feast-data\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.173407 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.173388 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-online-tls\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.173523 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.173505 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-registry-tls\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.179396 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.179364 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bft6\" (UniqueName: \"kubernetes.io/projected/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-kube-api-access-2bft6\") pod \"feast-credit-scoring-7bc47dd8b9-xfpxk\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.265574 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.265539 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:26.405651 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.405625 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk"] Jun 08 11:39:26.407907 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:39:26.407873 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb5a57c_20ae_4717_9673_8ae4b8bb80cd.slice/crio-f011ca13fc8f7dfbce52d823dd1f4c3c158639fafb8cff2b7dce7651eed922a5 WatchSource:0}: Error finding container f011ca13fc8f7dfbce52d823dd1f4c3c158639fafb8cff2b7dce7651eed922a5: Status 404 returned error can't find the container with id f011ca13fc8f7dfbce52d823dd1f4c3c158639fafb8cff2b7dce7651eed922a5 Jun 08 11:39:26.760758 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.760718 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" event={"ID":"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd","Type":"ContainerStarted","Data":"d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7"} Jun 08 11:39:26.760941 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:26.760776 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" event={"ID":"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd","Type":"ContainerStarted","Data":"f011ca13fc8f7dfbce52d823dd1f4c3c158639fafb8cff2b7dce7651eed922a5"} Jun 08 11:39:27.766794 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:27.766753 2569 generic.go:356] "Generic (PLEG): container finished" podID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerID="d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7" exitCode=0 Jun 08 11:39:27.767180 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:27.766847 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" event={"ID":"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd","Type":"ContainerDied","Data":"d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7"} Jun 08 11:39:28.775544 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:28.775511 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" event={"ID":"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd","Type":"ContainerStarted","Data":"ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0"} Jun 08 11:39:35.806655 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:35.806602 2569 generic.go:356] "Generic (PLEG): container finished" podID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerID="ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0" exitCode=0 Jun 08 11:39:35.806655 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:35.806639 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" event={"ID":"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd","Type":"ContainerDied","Data":"ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0"} Jun 08 11:39:36.815117 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:36.815075 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" event={"ID":"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd","Type":"ContainerStarted","Data":"c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb"} Jun 08 11:39:36.815628 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:36.815144 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" event={"ID":"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd","Type":"ContainerStarted","Data":"8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62"} Jun 08 11:39:36.838583 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:36.838530 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podStartSLOduration=11.838513294 podStartE2EDuration="11.838513294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:36.836513079 +0000 UTC m=+1166.320911997" watchObservedRunningTime="2026-06-08 11:39:36.838513294 +0000 UTC m=+1166.322912157" Jun 08 11:39:38.266786 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:38.266743 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:38.267158 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:38.266810 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:38.268766 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:38.268735 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jun 08 11:39:38.268914 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:38.268742 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Jun 08 11:39:41.266505 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:41.266455 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Jun 08 11:39:41.267056 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:41.266457 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jun 08 11:39:44.266042 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:44.265986 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Jun 08 11:39:44.271493 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:44.271462 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:44.271960 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:44.271933 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:44.274577 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:44.274553 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:46.266133 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:46.266098 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:47.266776 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:47.266742 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:47.864982 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:47.864957 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:39:51.783635 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.783581 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx"] Jun 08 11:39:51.787562 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.787545 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:51.789907 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.789886 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-5r9d5\"" Jun 08 11:39:51.790048 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.789948 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jun 08 11:39:51.790048 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.789973 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jun 08 11:39:51.790048 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.790007 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jun 08 11:39:51.796747 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.796719 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/debc6dc5-1a16-4783-b517-f021bc8388ea-feast-data\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:51.796875 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.796763 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-registry-tls\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:51.796955 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.796936 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:51.797005 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.796972 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6lbd\" (UniqueName: \"kubernetes.io/projected/debc6dc5-1a16-4783-b517-f021bc8388ea-kube-api-access-q6lbd\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:51.797299 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.797279 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx"] Jun 08 11:39:51.897805 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.897773 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/debc6dc5-1a16-4783-b517-f021bc8388ea-feast-data\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:51.897805 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.897804 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-registry-tls\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:51.898037 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.897871 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:51.898037 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.897889 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6lbd\" (UniqueName: \"kubernetes.io/projected/debc6dc5-1a16-4783-b517-f021bc8388ea-kube-api-access-q6lbd\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:51.898037 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:39:51.898018 2569 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Jun 08 11:39:51.898145 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:39:51.898087 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls podName:debc6dc5-1a16-4783-b517-f021bc8388ea nodeName:}" failed. No retries permitted until 2026-06-08 11:39:52.3980665 +0000 UTC m=+1181.882465342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls") pod "feast-driver-ranking-6b9f5fb8bb-qvjqx" (UID: "debc6dc5-1a16-4783-b517-f021bc8388ea") : secret "feast-driver-ranking-online-tls" not found Jun 08 11:39:51.898211 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.898191 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/debc6dc5-1a16-4783-b517-f021bc8388ea-feast-data\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:51.900399 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.900375 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-registry-tls\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:51.906602 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:51.906577 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6lbd\" (UniqueName: \"kubernetes.io/projected/debc6dc5-1a16-4783-b517-f021bc8388ea-kube-api-access-q6lbd\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:52.401088 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:52.401054 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:52.401293 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:39:52.401190 2569 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Jun 08 11:39:52.401293 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:39:52.401258 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls podName:debc6dc5-1a16-4783-b517-f021bc8388ea nodeName:}" failed. No retries permitted until 2026-06-08 11:39:53.401243783 +0000 UTC m=+1182.885642627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls") pod "feast-driver-ranking-6b9f5fb8bb-qvjqx" (UID: "debc6dc5-1a16-4783-b517-f021bc8388ea") : secret "feast-driver-ranking-online-tls" not found Jun 08 11:39:53.410011 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:53.409972 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:53.412526 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:53.412506 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls\") pod \"feast-driver-ranking-6b9f5fb8bb-qvjqx\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:53.597590 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:53.597547 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:39:53.734397 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:53.734369 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx"] Jun 08 11:39:53.735820 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:39:53.735789 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebc6dc5_1a16_4783_b517_f021bc8388ea.slice/crio-de84de16e489678c661f485f27c093ad22297b527aaf27db71e548e5513c0e3e WatchSource:0}: Error finding container de84de16e489678c661f485f27c093ad22297b527aaf27db71e548e5513c0e3e: Status 404 returned error can't find the container with id de84de16e489678c661f485f27c093ad22297b527aaf27db71e548e5513c0e3e Jun 08 11:39:53.896083 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:53.896050 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" event={"ID":"debc6dc5-1a16-4783-b517-f021bc8388ea","Type":"ContainerStarted","Data":"5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3"} Jun 08 11:39:53.896251 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:53.896101 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" event={"ID":"debc6dc5-1a16-4783-b517-f021bc8388ea","Type":"ContainerStarted","Data":"de84de16e489678c661f485f27c093ad22297b527aaf27db71e548e5513c0e3e"} Jun 08 11:39:58.919178 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:58.919142 2569 generic.go:356] "Generic (PLEG): container finished" podID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerID="5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3" exitCode=0 Jun 08 11:39:58.919637 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:58.919218 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" event={"ID":"debc6dc5-1a16-4783-b517-f021bc8388ea","Type":"ContainerDied","Data":"5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3"} Jun 08 11:39:59.925709 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:39:59.925675 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" event={"ID":"debc6dc5-1a16-4783-b517-f021bc8388ea","Type":"ContainerStarted","Data":"fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1"} Jun 08 11:40:06.958424 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:06.958394 2569 generic.go:356] "Generic (PLEG): container finished" podID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerID="fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1" exitCode=0 Jun 08 11:40:06.958823 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:06.958443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" event={"ID":"debc6dc5-1a16-4783-b517-f021bc8388ea","Type":"ContainerDied","Data":"fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1"} Jun 08 11:40:07.966179 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:07.966137 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" event={"ID":"debc6dc5-1a16-4783-b517-f021bc8388ea","Type":"ContainerStarted","Data":"a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c"} Jun 08 11:40:07.966699 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:07.966206 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" event={"ID":"debc6dc5-1a16-4783-b517-f021bc8388ea","Type":"ContainerStarted","Data":"8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49"} Jun 08 11:40:07.988467 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:07.988409 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podStartSLOduration=16.98839255 podStartE2EDuration="16.98839255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:40:07.985590532 +0000 UTC m=+1197.469989394" watchObservedRunningTime="2026-06-08 11:40:07.98839255 +0000 UTC m=+1197.472791413" Jun 08 11:40:08.598005 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:08.597960 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:40:08.598186 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:08.598070 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:40:08.599821 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:08.599782 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Jun 08 11:40:08.599968 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:08.599839 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jun 08 11:40:11.598221 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:11.598178 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Jun 08 11:40:11.598598 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:11.598180 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jun 08 11:40:13.598700 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:13.598665 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:40:13.599182 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:13.598772 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:40:14.598759 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:14.598705 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Jun 08 11:40:14.604214 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:14.604182 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:40:15.003796 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:15.003753 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:40:17.598152 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:17.598083 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Jun 08 11:40:20.598582 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:20.598546 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:40:20.600847 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:40:20.599179 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:41:37.504795 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.504762 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss"] Jun 08 11:41:37.505280 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.504964 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" podUID="60f9c948-6ae1-44ed-acf7-ed4f3d11e636" containerName="manager" containerID="cri-o://d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74" gracePeriod=10 Jun 08 11:41:37.786753 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.786719 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" Jun 08 11:41:37.804810 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.804743 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk"] Jun 08 11:41:37.805389 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.805284 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="registry" containerID="cri-o://8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62" gracePeriod=30 Jun 08 11:41:37.806094 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.805930 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="online" containerID="cri-o://c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb" gracePeriod=30 Jun 08 11:41:37.811234 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.811215 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx"] Jun 08 11:41:37.811771 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.811747 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="registry" containerID="cri-o://8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49" gracePeriod=30 Jun 08 11:41:37.812095 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.811812 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="online" containerID="cri-o://a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c" gracePeriod=30 Jun 08 11:41:37.834350 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.834313 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pd7f\" (UniqueName: \"kubernetes.io/projected/60f9c948-6ae1-44ed-acf7-ed4f3d11e636-kube-api-access-9pd7f\") pod \"60f9c948-6ae1-44ed-acf7-ed4f3d11e636\" (UID: \"60f9c948-6ae1-44ed-acf7-ed4f3d11e636\") " Jun 08 11:41:37.836699 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.836664 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f9c948-6ae1-44ed-acf7-ed4f3d11e636-kube-api-access-9pd7f" (OuterVolumeSpecName: "kube-api-access-9pd7f") pod "60f9c948-6ae1-44ed-acf7-ed4f3d11e636" (UID: "60f9c948-6ae1-44ed-acf7-ed4f3d11e636"). InnerVolumeSpecName "kube-api-access-9pd7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:41:37.935544 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:37.935500 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pd7f\" (UniqueName: \"kubernetes.io/projected/60f9c948-6ae1-44ed-acf7-ed4f3d11e636-kube-api-access-9pd7f\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:41:38.374882 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:38.374844 2569 generic.go:356] "Generic (PLEG): container finished" podID="60f9c948-6ae1-44ed-acf7-ed4f3d11e636" containerID="d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74" exitCode=0 Jun 08 11:41:38.374882 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:38.374884 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" event={"ID":"60f9c948-6ae1-44ed-acf7-ed4f3d11e636","Type":"ContainerDied","Data":"d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74"} Jun 08 11:41:38.374882 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:38.374889 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" Jun 08 11:41:38.375392 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:38.374930 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss" event={"ID":"60f9c948-6ae1-44ed-acf7-ed4f3d11e636","Type":"ContainerDied","Data":"0560e50da7596539d0e2de6b0a7fec322961480c435e126c6610fd4d82edb4f8"} Jun 08 11:41:38.375392 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:38.374953 2569 scope.go:117] "RemoveContainer" containerID="d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74" Jun 08 11:41:38.389048 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:38.388961 2569 scope.go:117] "RemoveContainer" containerID="d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74" Jun 08 11:41:38.389449 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:41:38.389401 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74\": container with ID starting with d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74 not found: ID does not exist" containerID="d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74" Jun 08 11:41:38.389559 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:38.389455 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74"} err="failed to get container status \"d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74\": rpc error: code = NotFound desc = could not find container \"d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74\": container with ID starting with d37ef81d304149f734d77ebbde6f94d3454952bc95134d026607a29dc968ed74 not found: ID does not exist" Jun 08 11:41:38.400488 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:38.400460 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss"] Jun 08 11:41:38.404018 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:38.403984 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5f5976bd8-5rkss"] Jun 08 11:41:38.953620 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:38.953532 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60f9c948-6ae1-44ed-acf7-ed4f3d11e636" path="/var/lib/kubelet/pods/60f9c948-6ae1-44ed-acf7-ed4f3d11e636/volumes" Jun 08 11:41:39.381187 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:39.381156 2569 generic.go:356] "Generic (PLEG): container finished" podID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerID="c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb" exitCode=0 Jun 08 11:41:39.381381 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:39.381230 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" event={"ID":"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd","Type":"ContainerDied","Data":"c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb"} Jun 08 11:41:39.383285 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:39.383264 2569 generic.go:356] "Generic (PLEG): container finished" podID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerID="a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c" exitCode=0 Jun 08 11:41:39.383422 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:39.383337 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" event={"ID":"debc6dc5-1a16-4783-b517-f021bc8388ea","Type":"ContainerDied","Data":"a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c"} Jun 08 11:41:44.272628 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:44.272583 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jun 08 11:41:45.001052 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:45.001010 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jun 08 11:41:54.273132 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:54.273081 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jun 08 11:41:55.001567 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:41:55.001523 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jun 08 11:42:04.272218 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:04.272172 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jun 08 11:42:04.272740 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:04.272298 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:42:05.001257 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:05.001213 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jun 08 11:42:05.001468 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:05.001356 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:42:07.865118 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:07.865081 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Jun 08 11:42:08.497930 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.497902 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:42:08.500799 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.500777 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:42:08.512454 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.512422 2569 generic.go:356] "Generic (PLEG): container finished" podID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerID="8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62" exitCode=137 Jun 08 11:42:08.512648 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.512479 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" Jun 08 11:42:08.512648 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.512508 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" event={"ID":"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd","Type":"ContainerDied","Data":"8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62"} Jun 08 11:42:08.512648 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.512564 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk" event={"ID":"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd","Type":"ContainerDied","Data":"f011ca13fc8f7dfbce52d823dd1f4c3c158639fafb8cff2b7dce7651eed922a5"} Jun 08 11:42:08.512648 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.512596 2569 scope.go:117] "RemoveContainer" containerID="c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb" Jun 08 11:42:08.514700 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.514674 2569 generic.go:356] "Generic (PLEG): container finished" podID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerID="8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49" exitCode=137 Jun 08 11:42:08.514815 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.514714 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" event={"ID":"debc6dc5-1a16-4783-b517-f021bc8388ea","Type":"ContainerDied","Data":"8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49"} Jun 08 11:42:08.514815 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.514741 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" Jun 08 11:42:08.514815 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.514756 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx" event={"ID":"debc6dc5-1a16-4783-b517-f021bc8388ea","Type":"ContainerDied","Data":"de84de16e489678c661f485f27c093ad22297b527aaf27db71e548e5513c0e3e"} Jun 08 11:42:08.521941 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.521916 2569 scope.go:117] "RemoveContainer" containerID="8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62" Jun 08 11:42:08.530571 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.530554 2569 scope.go:117] "RemoveContainer" containerID="ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0" Jun 08 11:42:08.539682 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.539661 2569 scope.go:117] "RemoveContainer" containerID="d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7" Jun 08 11:42:08.549304 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.549284 2569 scope.go:117] "RemoveContainer" containerID="c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb" Jun 08 11:42:08.549594 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:42:08.549577 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb\": container with ID starting with c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb not found: ID does not exist" containerID="c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb" Jun 08 11:42:08.549676 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.549629 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb"} err="failed to get container status \"c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb\": rpc error: code = NotFound desc = could not find container \"c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb\": container with ID starting with c07b078f6e440d921ee33cc324103e563ae5aa9fa651a1bf5faaf287d19a66bb not found: ID does not exist" Jun 08 11:42:08.549676 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.549648 2569 scope.go:117] "RemoveContainer" containerID="8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62" Jun 08 11:42:08.549901 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:42:08.549883 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62\": container with ID starting with 8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62 not found: ID does not exist" containerID="8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62" Jun 08 11:42:08.549956 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.549907 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62"} err="failed to get container status \"8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62\": rpc error: code = NotFound desc = could not find container \"8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62\": container with ID starting with 8f2df3c2405e72bdf0fbbc88bfcb58ef8d92786f3081ba8300ff08b9c22e8f62 not found: ID does not exist" Jun 08 11:42:08.549956 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.549924 2569 scope.go:117] "RemoveContainer" containerID="ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0" Jun 08 11:42:08.550141 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:42:08.550125 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0\": container with ID starting with ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0 not found: ID does not exist" containerID="ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0" Jun 08 11:42:08.550203 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.550145 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0"} err="failed to get container status \"ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0\": rpc error: code = NotFound desc = could not find container \"ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0\": container with ID starting with ceb0b5d0203fb6511b6c40f88e69906e3e82a7aab11d2e8f31cd1a89e62b62c0 not found: ID does not exist" Jun 08 11:42:08.550203 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.550159 2569 scope.go:117] "RemoveContainer" containerID="d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7" Jun 08 11:42:08.550441 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:42:08.550420 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7\": container with ID starting with d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7 not found: ID does not exist" containerID="d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7" Jun 08 11:42:08.550508 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.550450 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7"} err="failed to get container status \"d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7\": rpc error: code = NotFound desc = could not find container \"d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7\": container with ID starting with d85bad8b03bdd2d2831f3a0cb7d68f56214d5633d40bb025cca7ac60f06e49e7 not found: ID does not exist" Jun 08 11:42:08.550508 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.550470 2569 scope.go:117] "RemoveContainer" containerID="a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c" Jun 08 11:42:08.558907 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.558889 2569 scope.go:117] "RemoveContainer" containerID="8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49" Jun 08 11:42:08.567169 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.567148 2569 scope.go:117] "RemoveContainer" containerID="fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1" Jun 08 11:42:08.574340 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.574320 2569 scope.go:117] "RemoveContainer" containerID="5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3" Jun 08 11:42:08.581388 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.581367 2569 scope.go:117] "RemoveContainer" containerID="a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c" Jun 08 11:42:08.581677 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:42:08.581653 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c\": container with ID starting with a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c not found: ID does not exist" containerID="a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c" Jun 08 11:42:08.581759 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.581686 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c"} err="failed to get container status \"a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c\": rpc error: code = NotFound desc = could not find container \"a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c\": container with ID starting with a2be2db05d390e5a17c4d36b24922163fb1a411c04207f6a62515e7a54e5ba1c not found: ID does not exist" Jun 08 11:42:08.581759 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.581711 2569 scope.go:117] "RemoveContainer" containerID="8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49" Jun 08 11:42:08.581961 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:42:08.581938 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49\": container with ID starting with 8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49 not found: ID does not exist" containerID="8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49" Jun 08 11:42:08.582025 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.581968 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49"} err="failed to get container status \"8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49\": rpc error: code = NotFound desc = could not find container \"8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49\": container with ID starting with 8c7cf662f6ce17573032fa6f534c39a16be2772dfcf14e9471198c87103e6c49 not found: ID does not exist" Jun 08 11:42:08.582025 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.581992 2569 scope.go:117] "RemoveContainer" containerID="fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1" Jun 08 11:42:08.582237 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:42:08.582213 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1\": container with ID starting with fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1 not found: ID does not exist" containerID="fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1" Jun 08 11:42:08.582301 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.582242 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1"} err="failed to get container status \"fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1\": rpc error: code = NotFound desc = could not find container \"fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1\": container with ID starting with fa01aae8b0ebe2ad44e76d7c0e27aa961c00bea48571adefe771bcdc0c3789b1 not found: ID does not exist" Jun 08 11:42:08.582301 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.582259 2569 scope.go:117] "RemoveContainer" containerID="5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3" Jun 08 11:42:08.582501 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:42:08.582484 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3\": container with ID starting with 5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3 not found: ID does not exist" containerID="5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3" Jun 08 11:42:08.582549 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.582507 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3"} err="failed to get container status \"5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3\": rpc error: code = NotFound desc = could not find container \"5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3\": container with ID starting with 5e84db2590a3b3bff7683772988ddfc08146a3f320e1b71d928cfede71ba4fe3 not found: ID does not exist" Jun 08 11:42:08.608251 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.608227 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls\") pod \"debc6dc5-1a16-4783-b517-f021bc8388ea\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " Jun 08 11:42:08.608375 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.608283 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/debc6dc5-1a16-4783-b517-f021bc8388ea-feast-data\") pod \"debc6dc5-1a16-4783-b517-f021bc8388ea\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " Jun 08 11:42:08.608375 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.608316 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bft6\" (UniqueName: \"kubernetes.io/projected/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-kube-api-access-2bft6\") pod \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " Jun 08 11:42:08.608375 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.608356 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6lbd\" (UniqueName: \"kubernetes.io/projected/debc6dc5-1a16-4783-b517-f021bc8388ea-kube-api-access-q6lbd\") pod \"debc6dc5-1a16-4783-b517-f021bc8388ea\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " Jun 08 11:42:08.608543 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.608397 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-registry-tls\") pod \"debc6dc5-1a16-4783-b517-f021bc8388ea\" (UID: \"debc6dc5-1a16-4783-b517-f021bc8388ea\") " Jun 08 11:42:08.608543 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.608447 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-registry-tls\") pod \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " Jun 08 11:42:08.608543 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.608471 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-online-tls\") pod \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " Jun 08 11:42:08.608543 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.608516 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-feast-data\") pod \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\" (UID: \"4fb5a57c-20ae-4717-9673-8ae4b8bb80cd\") " Jun 08 11:42:08.609198 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.609166 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debc6dc5-1a16-4783-b517-f021bc8388ea-feast-data" (OuterVolumeSpecName: "feast-data") pod "debc6dc5-1a16-4783-b517-f021bc8388ea" (UID: "debc6dc5-1a16-4783-b517-f021bc8388ea"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:42:08.611226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.611191 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-online-tls" (OuterVolumeSpecName: "online-tls") pod "4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" (UID: "4fb5a57c-20ae-4717-9673-8ae4b8bb80cd"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:42:08.611324 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.611231 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls" (OuterVolumeSpecName: "online-tls") pod "debc6dc5-1a16-4783-b517-f021bc8388ea" (UID: "debc6dc5-1a16-4783-b517-f021bc8388ea"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:42:08.611324 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.611239 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debc6dc5-1a16-4783-b517-f021bc8388ea-kube-api-access-q6lbd" (OuterVolumeSpecName: "kube-api-access-q6lbd") pod "debc6dc5-1a16-4783-b517-f021bc8388ea" (UID: "debc6dc5-1a16-4783-b517-f021bc8388ea"). InnerVolumeSpecName "kube-api-access-q6lbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:42:08.611324 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.611255 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "debc6dc5-1a16-4783-b517-f021bc8388ea" (UID: "debc6dc5-1a16-4783-b517-f021bc8388ea"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:42:08.611324 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.611310 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-kube-api-access-2bft6" (OuterVolumeSpecName: "kube-api-access-2bft6") pod "4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" (UID: "4fb5a57c-20ae-4717-9673-8ae4b8bb80cd"). InnerVolumeSpecName "kube-api-access-2bft6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:42:08.611481 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.611404 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" (UID: "4fb5a57c-20ae-4717-9673-8ae4b8bb80cd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:42:08.618455 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.618434 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-feast-data" (OuterVolumeSpecName: "feast-data") pod "4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" (UID: "4fb5a57c-20ae-4717-9673-8ae4b8bb80cd"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:42:08.709781 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.709748 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/debc6dc5-1a16-4783-b517-f021bc8388ea-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:42:08.709781 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.709776 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bft6\" (UniqueName: \"kubernetes.io/projected/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-kube-api-access-2bft6\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:42:08.709781 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.709787 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6lbd\" (UniqueName: \"kubernetes.io/projected/debc6dc5-1a16-4783-b517-f021bc8388ea-kube-api-access-q6lbd\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:42:08.710004 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.709796 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:42:08.710004 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.709805 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:42:08.710004 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.709813 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:42:08.710004 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.709821 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:42:08.710004 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.709829 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/debc6dc5-1a16-4783-b517-f021bc8388ea-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:42:08.846246 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.846218 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk"] Jun 08 11:42:08.850127 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.850101 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7bc47dd8b9-xfpxk"] Jun 08 11:42:08.860972 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.860945 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx"] Jun 08 11:42:08.864211 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.864186 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6b9f5fb8bb-qvjqx"] Jun 08 11:42:08.953399 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.953368 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" path="/var/lib/kubelet/pods/4fb5a57c-20ae-4717-9673-8ae4b8bb80cd/volumes" Jun 08 11:42:08.953891 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:08.953876 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" path="/var/lib/kubelet/pods/debc6dc5-1a16-4783-b517-f021bc8388ea/volumes" Jun 08 11:42:10.636340 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636308 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt"] Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636657 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="online" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636673 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="online" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636684 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="feast-init" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636689 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="feast-init" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636697 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="feast-init" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636702 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="feast-init" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636709 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="registry" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636714 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="registry" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636721 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="online" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636726 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="online" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636744 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="registry" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636749 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="registry" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636754 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="feast-apply" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636760 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="feast-apply" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636767 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f9c948-6ae1-44ed-acf7-ed4f3d11e636" containerName="manager" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636772 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f9c948-6ae1-44ed-acf7-ed4f3d11e636" containerName="manager" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636780 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="feast-apply" Jun 08 11:42:10.636829 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636785 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="feast-apply" Jun 08 11:42:10.637362 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636842 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="online" Jun 08 11:42:10.637362 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636850 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="online" Jun 08 11:42:10.637362 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636857 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f9c948-6ae1-44ed-acf7-ed4f3d11e636" containerName="manager" Jun 08 11:42:10.637362 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636864 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fb5a57c-20ae-4717-9673-8ae4b8bb80cd" containerName="registry" Jun 08 11:42:10.637362 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.636870 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="debc6dc5-1a16-4783-b517-f021bc8388ea" containerName="registry" Jun 08 11:42:10.642091 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.642071 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.644859 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.644838 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 08 11:42:10.644970 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.644847 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 08 11:42:10.645671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.645653 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-47dm6\"" Jun 08 11:42:10.645777 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.645675 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 08 11:42:10.645777 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.645683 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 08 11:42:10.645777 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.645725 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 08 11:42:10.645777 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.645653 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 08 11:42:10.653043 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.653018 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt"] Jun 08 11:42:10.829062 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.829011 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.829062 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.829064 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.829336 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.829118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.829336 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.829137 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.829336 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.829153 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2s84\" (UniqueName: \"kubernetes.io/projected/5a685e31-9a8c-4f53-9735-4dcc915df894-kube-api-access-k2s84\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.829336 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.829254 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5a685e31-9a8c-4f53-9735-4dcc915df894-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.930354 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.930261 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5a685e31-9a8c-4f53-9735-4dcc915df894-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.930354 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.930311 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.930354 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.930330 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.930656 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.930362 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.930656 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.930378 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.930656 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.930393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2s84\" (UniqueName: \"kubernetes.io/projected/5a685e31-9a8c-4f53-9735-4dcc915df894-kube-api-access-k2s84\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.930767 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.930658 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5a685e31-9a8c-4f53-9735-4dcc915df894-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.932954 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.932935 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 08 11:42:10.933035 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.932961 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 08 11:42:10.933035 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.933011 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 08 11:42:10.933114 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.933051 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 08 11:42:10.937518 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.937504 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 08 11:42:10.943796 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.943767 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.943929 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.943894 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.943929 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.943919 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.944026 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.943962 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:10.948771 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.948746 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 08 11:42:10.961749 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:10.961721 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2s84\" (UniqueName: \"kubernetes.io/projected/5a685e31-9a8c-4f53-9735-4dcc915df894-kube-api-access-k2s84\") pod \"feast-simple-feast-setup-75c64c888b-kg8xt\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:11.262792 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:11.262758 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-47dm6\"" Jun 08 11:42:11.271383 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:11.271356 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:11.416402 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:11.416351 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt"] Jun 08 11:42:11.418488 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:42:11.418460 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a685e31_9a8c_4f53_9735_4dcc915df894.slice/crio-526d36646104c887f05d057be9cdef224f2988b7b62e88ed1df54243b42fb279 WatchSource:0}: Error finding container 526d36646104c887f05d057be9cdef224f2988b7b62e88ed1df54243b42fb279: Status 404 returned error can't find the container with id 526d36646104c887f05d057be9cdef224f2988b7b62e88ed1df54243b42fb279 Jun 08 11:42:11.420347 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:11.420328 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 08 11:42:11.530393 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:11.530310 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerStarted","Data":"526d36646104c887f05d057be9cdef224f2988b7b62e88ed1df54243b42fb279"} Jun 08 11:42:25.611587 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:25.611487 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerStarted","Data":"038378de1aacb4dbe2ad642cafaadf3bbc9b9a3af80cd3dc0afc915a9d99aa1b"} Jun 08 11:42:30.637970 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:30.637934 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerID="038378de1aacb4dbe2ad642cafaadf3bbc9b9a3af80cd3dc0afc915a9d99aa1b" exitCode=0 Jun 08 11:42:30.638354 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:30.638008 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerDied","Data":"038378de1aacb4dbe2ad642cafaadf3bbc9b9a3af80cd3dc0afc915a9d99aa1b"} Jun 08 11:42:31.644100 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:31.644062 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerStarted","Data":"f084af4a287c27ac2f57db11881ce3731a15d03d7a8a1dd0b51c4f74ec1a2a8d"} Jun 08 11:42:36.664762 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:36.664728 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerID="f084af4a287c27ac2f57db11881ce3731a15d03d7a8a1dd0b51c4f74ec1a2a8d" exitCode=0 Jun 08 11:42:36.665146 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:36.664798 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerDied","Data":"f084af4a287c27ac2f57db11881ce3731a15d03d7a8a1dd0b51c4f74ec1a2a8d"} Jun 08 11:42:37.682426 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:37.682385 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerStarted","Data":"f1463c6599deae88b9d1742397a27e8af76d1540b4db0cb234fce8e8b76bbadc"} Jun 08 11:42:37.682830 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:37.682449 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerStarted","Data":"9e667ed1a4290f1e1c8d46d434f985662054c6bba1e05ec5fa21b64ceae4d7c4"} Jun 08 11:42:37.682830 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:37.682469 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerStarted","Data":"7338dc85d9fc5b8040fd522f40ef3c0b556b87fba255d77cb597524b6f0d21e9"} Jun 08 11:42:37.682830 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:37.682486 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerStarted","Data":"f0778f778bae697a8d6328ccf48de2cacb53f639ae7b14ec3acc70b54d987ad1"} Jun 08 11:42:37.707483 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:37.707422 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podStartSLOduration=13.77032228 podStartE2EDuration="27.707398347s" totalImagesPullingTime="13.937076067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:42:37.703915866 +0000 UTC m=+1347.188314731" watchObservedRunningTime="2026-06-08 11:42:37.707398347 +0000 UTC m=+1347.191797212" Jun 08 11:42:38.272885 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:38.272653 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:38.272885 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:38.272830 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:38.272885 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:38.272871 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:38.272885 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:38.272886 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:38.275021 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:38.274971 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Jun 08 11:42:38.275021 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:38.275007 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused" Jun 08 11:42:38.275251 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:38.275052 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Jun 08 11:42:38.275251 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:38.275156 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.32:8816: connect: connection refused" Jun 08 11:42:41.272799 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:41.272372 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:41.272799 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:41.272441 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:41.272799 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:41.272462 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:41.272799 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:41.272659 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.32:8816: connect: connection refused" Jun 08 11:42:41.272799 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:41.272659 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Jun 08 11:42:41.272799 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:41.272742 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Jun 08 11:42:41.273580 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:41.272930 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:41.273580 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:41.272997 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused" Jun 08 11:42:44.273034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:44.272998 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:44.273630 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:44.273361 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:44.273630 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:44.273542 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:44.274327 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:44.274302 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:44.274417 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:44.274382 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:44.274456 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:44.274435 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:44.277730 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:44.277713 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:42:44.724619 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:42:44.724592 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:43:45.984182 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:45.984142 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt"] Jun 08 11:43:45.984766 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:45.984425 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="registry" containerID="cri-o://f0778f778bae697a8d6328ccf48de2cacb53f639ae7b14ec3acc70b54d987ad1" gracePeriod=30 Jun 08 11:43:45.984766 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:45.984576 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="offline" containerID="cri-o://9e667ed1a4290f1e1c8d46d434f985662054c6bba1e05ec5fa21b64ceae4d7c4" gracePeriod=30 Jun 08 11:43:45.984766 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:45.984575 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="ui" containerID="cri-o://f1463c6599deae88b9d1742397a27e8af76d1540b4db0cb234fce8e8b76bbadc" gracePeriod=30 Jun 08 11:43:45.984766 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:45.984737 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="online" containerID="cri-o://7338dc85d9fc5b8040fd522f40ef3c0b556b87fba255d77cb597524b6f0d21e9" gracePeriod=30 Jun 08 11:43:46.336770 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.336688 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4"] Jun 08 11:43:46.339802 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.339785 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.342229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.342203 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-m8l8q\"" Jun 08 11:43:46.351537 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.351510 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4"] Jun 08 11:43:46.468438 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.468401 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.468668 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.468453 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.468668 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.468471 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h52vd\" (UniqueName: \"kubernetes.io/projected/54fc71dc-cb96-450b-ba97-2b3a87fe0550-kube-api-access-h52vd\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.468668 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.468570 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.468668 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.468637 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/54fc71dc-cb96-450b-ba97-2b3a87fe0550-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.468668 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.468668 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.569505 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.569467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/54fc71dc-cb96-450b-ba97-2b3a87fe0550-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.569720 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.569522 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.569720 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.569560 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.569720 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.569597 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.569720 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.569650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h52vd\" (UniqueName: \"kubernetes.io/projected/54fc71dc-cb96-450b-ba97-2b3a87fe0550-kube-api-access-h52vd\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.569720 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.569697 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.569986 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:43:46.569833 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 08 11:43:46.569986 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.569874 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/54fc71dc-cb96-450b-ba97-2b3a87fe0550-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.569986 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:43:46.569895 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-registry-tls podName:54fc71dc-cb96-450b-ba97-2b3a87fe0550 nodeName:}" failed. No retries permitted until 2026-06-08 11:43:47.069875627 +0000 UTC m=+1416.554274490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-registry-tls") pod "feast-simple-feast-setup-75c64c888b-xhdg4" (UID: "54fc71dc-cb96-450b-ba97-2b3a87fe0550") : secret "feast-simple-feast-setup-registry-tls" not found Jun 08 11:43:46.572284 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.572254 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.572418 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.572375 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.572504 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.572483 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.577693 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.577664 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h52vd\" (UniqueName: \"kubernetes.io/projected/54fc71dc-cb96-450b-ba97-2b3a87fe0550-kube-api-access-h52vd\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:46.997553 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.997521 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerID="f1463c6599deae88b9d1742397a27e8af76d1540b4db0cb234fce8e8b76bbadc" exitCode=0 Jun 08 11:43:46.997553 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.997547 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerID="7338dc85d9fc5b8040fd522f40ef3c0b556b87fba255d77cb597524b6f0d21e9" exitCode=0 Jun 08 11:43:46.997967 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.997593 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerDied","Data":"f1463c6599deae88b9d1742397a27e8af76d1540b4db0cb234fce8e8b76bbadc"} Jun 08 11:43:46.997967 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:46.997648 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerDied","Data":"7338dc85d9fc5b8040fd522f40ef3c0b556b87fba255d77cb597524b6f0d21e9"} Jun 08 11:43:47.074241 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:47.074208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:47.076760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:47.076730 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-xhdg4\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:47.250854 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:47.250763 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:47.385835 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:47.385810 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4"] Jun 08 11:43:47.388402 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:43:47.388369 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54fc71dc_cb96_450b_ba97_2b3a87fe0550.slice/crio-d05e9d1121e85cba1e18e5be3a1738f97c9d276efc1f94e4e67f15b151fffeb5 WatchSource:0}: Error finding container d05e9d1121e85cba1e18e5be3a1738f97c9d276efc1f94e4e67f15b151fffeb5: Status 404 returned error can't find the container with id d05e9d1121e85cba1e18e5be3a1738f97c9d276efc1f94e4e67f15b151fffeb5 Jun 08 11:43:48.003508 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:48.003476 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerStarted","Data":"41fcc99107750b0c69f9f41fc7d5d594571c39cfe26fe3a6be1e1616e0a4c3d2"} Jun 08 11:43:48.003907 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:48.003526 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerStarted","Data":"d05e9d1121e85cba1e18e5be3a1738f97c9d276efc1f94e4e67f15b151fffeb5"} Jun 08 11:43:52.020695 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:52.020662 2569 generic.go:356] "Generic (PLEG): container finished" podID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerID="41fcc99107750b0c69f9f41fc7d5d594571c39cfe26fe3a6be1e1616e0a4c3d2" exitCode=0 Jun 08 11:43:52.021025 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:52.020724 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerDied","Data":"41fcc99107750b0c69f9f41fc7d5d594571c39cfe26fe3a6be1e1616e0a4c3d2"} Jun 08 11:43:53.027575 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:53.027532 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerStarted","Data":"111082447b371602b3139c006fc74491fd09d1e0b0d2019a1e38b568757ca65f"} Jun 08 11:43:54.274677 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:54.274603 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused" Jun 08 11:43:54.722017 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:54.721967 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Jun 08 11:43:58.052342 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:58.052309 2569 generic.go:356] "Generic (PLEG): container finished" podID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerID="111082447b371602b3139c006fc74491fd09d1e0b0d2019a1e38b568757ca65f" exitCode=0 Jun 08 11:43:58.052731 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:58.052361 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerDied","Data":"111082447b371602b3139c006fc74491fd09d1e0b0d2019a1e38b568757ca65f"} Jun 08 11:43:59.069300 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.069252 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerStarted","Data":"3511430e79f989fd9d8f0418e9b512872a8cdb193f2f8e86f95666a90a7350f8"} Jun 08 11:43:59.069856 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.069319 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerStarted","Data":"aa9558bc5eeded39697997acbd5fc182f33a1f8b375487eb51cc05617458f395"} Jun 08 11:43:59.069856 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.069339 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerStarted","Data":"a9742f8e5f3bae92f73a73c2af83c289d0d971cdbbb6e394f94727eaa5ca6934"} Jun 08 11:43:59.069856 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.069353 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerStarted","Data":"0f6d06e90d42e472ec0e865bbe20d48e63901a8d9d4fae15f12ff126055652dc"} Jun 08 11:43:59.095100 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.095025 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podStartSLOduration=13.095003572 podStartE2EDuration="13.095003572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:43:59.092968839 +0000 UTC m=+1428.577367714" watchObservedRunningTime="2026-06-08 11:43:59.095003572 +0000 UTC m=+1428.579402436" Jun 08 11:43:59.251757 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.251704 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:59.251945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.251779 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:59.251945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.251801 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:59.251945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.251821 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:43:59.254139 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.254086 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Jun 08 11:43:59.254297 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.254178 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Jun 08 11:43:59.254297 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.254178 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.33:8816: connect: connection refused" Jun 08 11:43:59.254297 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:43:59.254278 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jun 08 11:44:02.251555 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:02.251499 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Jun 08 11:44:02.252046 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:02.251589 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jun 08 11:44:02.252046 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:02.251521 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Jun 08 11:44:02.252046 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:02.251514 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.33:8816: connect: connection refused" Jun 08 11:44:04.274421 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:04.274376 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused" Jun 08 11:44:04.722174 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:04.722135 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Jun 08 11:44:05.251843 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:05.251812 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:05.252092 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:05.252075 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:05.252321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:05.252310 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:05.252411 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:05.252401 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:05.252574 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:05.252554 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:05.256381 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:05.256312 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:06.118752 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:06.118721 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:06.119188 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:06.118773 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:06.119188 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:06.118791 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:06.119263 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:06.119229 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:06.121696 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:06.121677 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:07.123789 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:07.123764 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:44:14.274328 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:14.274280 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused" Jun 08 11:44:14.274752 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:14.274449 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:44:14.722276 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:14.722230 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Jun 08 11:44:14.722472 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:14.722385 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:44:16.162817 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.162786 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerID="9e667ed1a4290f1e1c8d46d434f985662054c6bba1e05ec5fa21b64ceae4d7c4" exitCode=137 Jun 08 11:44:16.162817 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.162810 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerID="f0778f778bae697a8d6328ccf48de2cacb53f639ae7b14ec3acc70b54d987ad1" exitCode=137 Jun 08 11:44:16.163203 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.162854 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerDied","Data":"9e667ed1a4290f1e1c8d46d434f985662054c6bba1e05ec5fa21b64ceae4d7c4"} Jun 08 11:44:16.163203 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.162892 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerDied","Data":"f0778f778bae697a8d6328ccf48de2cacb53f639ae7b14ec3acc70b54d987ad1"} Jun 08 11:44:16.639219 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.639193 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:44:16.750210 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.750175 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-registry-tls\") pod \"5a685e31-9a8c-4f53-9735-4dcc915df894\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " Jun 08 11:44:16.750210 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.750215 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2s84\" (UniqueName: \"kubernetes.io/projected/5a685e31-9a8c-4f53-9735-4dcc915df894-kube-api-access-k2s84\") pod \"5a685e31-9a8c-4f53-9735-4dcc915df894\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " Jun 08 11:44:16.750450 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.750233 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-ui-tls\") pod \"5a685e31-9a8c-4f53-9735-4dcc915df894\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " Jun 08 11:44:16.750450 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.750271 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5a685e31-9a8c-4f53-9735-4dcc915df894-feast-data\") pod \"5a685e31-9a8c-4f53-9735-4dcc915df894\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " Jun 08 11:44:16.750450 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.750405 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-online-tls\") pod \"5a685e31-9a8c-4f53-9735-4dcc915df894\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " Jun 08 11:44:16.750605 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.750466 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-offline-tls\") pod \"5a685e31-9a8c-4f53-9735-4dcc915df894\" (UID: \"5a685e31-9a8c-4f53-9735-4dcc915df894\") " Jun 08 11:44:16.750848 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.750785 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a685e31-9a8c-4f53-9735-4dcc915df894-feast-data" (OuterVolumeSpecName: "feast-data") pod "5a685e31-9a8c-4f53-9735-4dcc915df894" (UID: "5a685e31-9a8c-4f53-9735-4dcc915df894"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:44:16.752793 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.752741 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-online-tls" (OuterVolumeSpecName: "online-tls") pod "5a685e31-9a8c-4f53-9735-4dcc915df894" (UID: "5a685e31-9a8c-4f53-9735-4dcc915df894"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:44:16.753005 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.752984 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5a685e31-9a8c-4f53-9735-4dcc915df894" (UID: "5a685e31-9a8c-4f53-9735-4dcc915df894"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:44:16.753108 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.753020 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a685e31-9a8c-4f53-9735-4dcc915df894-kube-api-access-k2s84" (OuterVolumeSpecName: "kube-api-access-k2s84") pod "5a685e31-9a8c-4f53-9735-4dcc915df894" (UID: "5a685e31-9a8c-4f53-9735-4dcc915df894"). InnerVolumeSpecName "kube-api-access-k2s84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:44:16.753175 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.753108 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5a685e31-9a8c-4f53-9735-4dcc915df894" (UID: "5a685e31-9a8c-4f53-9735-4dcc915df894"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:44:16.753175 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.753123 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5a685e31-9a8c-4f53-9735-4dcc915df894" (UID: "5a685e31-9a8c-4f53-9735-4dcc915df894"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:44:16.852136 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.852100 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5a685e31-9a8c-4f53-9735-4dcc915df894-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:44:16.852136 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.852129 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:44:16.852136 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.852138 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-offline-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:44:16.852136 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.852148 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:44:16.852398 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.852156 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2s84\" (UniqueName: \"kubernetes.io/projected/5a685e31-9a8c-4f53-9735-4dcc915df894-kube-api-access-k2s84\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:44:16.852398 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:16.852164 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5a685e31-9a8c-4f53-9735-4dcc915df894-ui-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:44:17.170163 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:17.170074 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" event={"ID":"5a685e31-9a8c-4f53-9735-4dcc915df894","Type":"ContainerDied","Data":"526d36646104c887f05d057be9cdef224f2988b7b62e88ed1df54243b42fb279"} Jun 08 11:44:17.170163 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:17.170112 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt" Jun 08 11:44:17.170163 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:17.170135 2569 scope.go:117] "RemoveContainer" containerID="f1463c6599deae88b9d1742397a27e8af76d1540b4db0cb234fce8e8b76bbadc" Jun 08 11:44:17.179485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:17.179463 2569 scope.go:117] "RemoveContainer" containerID="9e667ed1a4290f1e1c8d46d434f985662054c6bba1e05ec5fa21b64ceae4d7c4" Jun 08 11:44:17.187598 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:17.187566 2569 scope.go:117] "RemoveContainer" containerID="7338dc85d9fc5b8040fd522f40ef3c0b556b87fba255d77cb597524b6f0d21e9" Jun 08 11:44:17.189805 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:17.189782 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt"] Jun 08 11:44:17.193213 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:17.193193 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kg8xt"] Jun 08 11:44:17.196243 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:17.196227 2569 scope.go:117] "RemoveContainer" containerID="f0778f778bae697a8d6328ccf48de2cacb53f639ae7b14ec3acc70b54d987ad1" Jun 08 11:44:17.203779 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:17.203761 2569 scope.go:117] "RemoveContainer" containerID="f084af4a287c27ac2f57db11881ce3731a15d03d7a8a1dd0b51c4f74ec1a2a8d" Jun 08 11:44:17.210931 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:17.210915 2569 scope.go:117] "RemoveContainer" containerID="038378de1aacb4dbe2ad642cafaadf3bbc9b9a3af80cd3dc0afc915a9d99aa1b" Jun 08 11:44:18.958806 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:44:18.958774 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" path="/var/lib/kubelet/pods/5a685e31-9a8c-4f53-9735-4dcc915df894/volumes" Jun 08 11:45:09.964885 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.964849 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw"] Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965163 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="feast-init" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965174 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="feast-init" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965182 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="online" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965189 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="online" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965202 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="feast-apply" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965207 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="feast-apply" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965214 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="registry" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965219 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="registry" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965228 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="offline" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965233 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="offline" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965243 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="ui" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965249 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="ui" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965298 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="offline" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965307 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="online" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965314 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="ui" Jun 08 11:45:09.965335 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.965319 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a685e31-9a8c-4f53-9735-4dcc915df894" containerName="registry" Jun 08 11:45:09.968734 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.968718 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:09.971282 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.971255 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 08 11:45:09.971455 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.971434 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 08 11:45:09.971601 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.971475 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 08 11:45:09.971601 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.971439 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 08 11:45:09.971601 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.971595 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-k7wcm\"" Jun 08 11:45:09.971800 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.971663 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 08 11:45:09.972192 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.972177 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 08 11:45:09.978805 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:09.978783 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw"] Jun 08 11:45:10.124275 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.124244 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.124464 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.124286 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8q4z\" (UniqueName: \"kubernetes.io/projected/cb323467-9b6d-4e87-a17f-354e9ab55e68-kube-api-access-x8q4z\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.124464 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.124344 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/cb323467-9b6d-4e87-a17f-354e9ab55e68-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.124464 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.124397 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.124464 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.124443 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb323467-9b6d-4e87-a17f-354e9ab55e68-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.124731 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.124554 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.225389 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.225361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.225560 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.225399 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb323467-9b6d-4e87-a17f-354e9ab55e68-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.225560 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.225475 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.225560 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:45:10.225520 2569 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 08 11:45:10.225701 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:45:10.225594 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-offline-tls podName:cb323467-9b6d-4e87-a17f-354e9ab55e68 nodeName:}" failed. No retries permitted until 2026-06-08 11:45:10.725573142 +0000 UTC m=+1500.209972004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-offline-tls") pod "feast-simple-feast-remote-setup-669bcf45fc-qsxtw" (UID: "cb323467-9b6d-4e87-a17f-354e9ab55e68") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 08 11:45:10.225701 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.225520 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.225701 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.225682 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8q4z\" (UniqueName: \"kubernetes.io/projected/cb323467-9b6d-4e87-a17f-354e9ab55e68-kube-api-access-x8q4z\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.225816 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.225720 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/cb323467-9b6d-4e87-a17f-354e9ab55e68-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.225901 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.225871 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb323467-9b6d-4e87-a17f-354e9ab55e68-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.226376 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.226290 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/cb323467-9b6d-4e87-a17f-354e9ab55e68-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.228059 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.228037 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.228195 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.228173 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.233706 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.233689 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8q4z\" (UniqueName: \"kubernetes.io/projected/cb323467-9b6d-4e87-a17f-354e9ab55e68-kube-api-access-x8q4z\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.729724 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.729690 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.732301 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.732277 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-qsxtw\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:10.880794 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.880764 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-k7wcm\"" Jun 08 11:45:10.888669 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:10.888649 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:11.027996 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:11.027970 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw"] Jun 08 11:45:11.030102 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:45:11.030074 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb323467_9b6d_4e87_a17f_354e9ab55e68.slice/crio-64a103029787e9da8204ced666d1de4288df01104c81847ee3886f7bae98cfd6 WatchSource:0}: Error finding container 64a103029787e9da8204ced666d1de4288df01104c81847ee3886f7bae98cfd6: Status 404 returned error can't find the container with id 64a103029787e9da8204ced666d1de4288df01104c81847ee3886f7bae98cfd6 Jun 08 11:45:11.403401 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:11.403306 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerStarted","Data":"d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2"} Jun 08 11:45:11.403401 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:11.403367 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerStarted","Data":"64a103029787e9da8204ced666d1de4288df01104c81847ee3886f7bae98cfd6"} Jun 08 11:45:16.426407 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:16.426371 2569 generic.go:356] "Generic (PLEG): container finished" podID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerID="d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2" exitCode=0 Jun 08 11:45:16.426905 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:16.426430 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerDied","Data":"d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2"} Jun 08 11:45:17.432260 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:17.432227 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerStarted","Data":"96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43"} Jun 08 11:45:22.456666 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:22.456561 2569 generic.go:356] "Generic (PLEG): container finished" podID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerID="96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43" exitCode=0 Jun 08 11:45:22.456666 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:22.456642 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerDied","Data":"96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43"} Jun 08 11:45:23.466590 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:23.466496 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerStarted","Data":"fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e"} Jun 08 11:45:23.466590 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:23.466567 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerStarted","Data":"b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041"} Jun 08 11:45:23.466590 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:23.466590 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerStarted","Data":"a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5"} Jun 08 11:45:23.492758 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:23.492688 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podStartSLOduration=14.492665344 podStartE2EDuration="14.492665344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:45:23.489754125 +0000 UTC m=+1512.974152989" watchObservedRunningTime="2026-06-08 11:45:23.492665344 +0000 UTC m=+1512.977064205" Jun 08 11:45:25.889288 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:25.889249 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:25.889818 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:25.889388 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:25.889818 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:25.889419 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:25.891368 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:25.891329 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Jun 08 11:45:25.891368 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:25.891347 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.34:8816: connect: connection refused" Jun 08 11:45:25.891568 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:25.891396 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Jun 08 11:45:28.890815 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:28.890781 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:28.891286 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:28.890913 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:28.891417 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:28.891388 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:28.891561 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:28.891436 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:28.891800 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:28.891777 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:28.892772 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:28.892753 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:28.895359 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:28.895337 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:29.497204 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:29.497171 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:45:29.500186 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:45:29.500156 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:46:31.374739 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:31.374705 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw"] Jun 08 11:46:31.375288 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:31.375001 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="online" containerID="cri-o://a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5" gracePeriod=30 Jun 08 11:46:31.375288 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:31.375086 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="ui" containerID="cri-o://fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e" gracePeriod=30 Jun 08 11:46:31.375288 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:31.375119 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="offline" containerID="cri-o://b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041" gracePeriod=30 Jun 08 11:46:31.535552 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:31.535514 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4"] Jun 08 11:46:31.535850 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:31.535820 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="registry" containerID="cri-o://0f6d06e90d42e472ec0e865bbe20d48e63901a8d9d4fae15f12ff126055652dc" gracePeriod=30 Jun 08 11:46:31.536069 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:31.535845 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="ui" containerID="cri-o://3511430e79f989fd9d8f0418e9b512872a8cdb193f2f8e86f95666a90a7350f8" gracePeriod=30 Jun 08 11:46:31.536178 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:31.535888 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="online" containerID="cri-o://a9742f8e5f3bae92f73a73c2af83c289d0d971cdbbb6e394f94727eaa5ca6934" gracePeriod=30 Jun 08 11:46:31.536241 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:31.535930 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="offline" containerID="cri-o://aa9558bc5eeded39697997acbd5fc182f33a1f8b375487eb51cc05617458f395" gracePeriod=30 Jun 08 11:46:32.763665 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:32.763627 2569 generic.go:356] "Generic (PLEG): container finished" podID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerID="fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e" exitCode=0 Jun 08 11:46:32.763665 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:32.763660 2569 generic.go:356] "Generic (PLEG): container finished" podID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerID="a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5" exitCode=0 Jun 08 11:46:32.764122 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:32.763633 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerDied","Data":"fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e"} Jun 08 11:46:32.764122 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:32.763750 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerDied","Data":"a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5"} Jun 08 11:46:32.766433 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:32.766412 2569 generic.go:356] "Generic (PLEG): container finished" podID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerID="3511430e79f989fd9d8f0418e9b512872a8cdb193f2f8e86f95666a90a7350f8" exitCode=0 Jun 08 11:46:32.766433 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:32.766429 2569 generic.go:356] "Generic (PLEG): container finished" podID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerID="a9742f8e5f3bae92f73a73c2af83c289d0d971cdbbb6e394f94727eaa5ca6934" exitCode=0 Jun 08 11:46:32.766583 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:32.766451 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerDied","Data":"3511430e79f989fd9d8f0418e9b512872a8cdb193f2f8e86f95666a90a7350f8"} Jun 08 11:46:32.766583 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:32.766477 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerDied","Data":"a9742f8e5f3bae92f73a73c2af83c289d0d971cdbbb6e394f94727eaa5ca6934"} Jun 08 11:46:35.252543 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:35.252503 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Jun 08 11:46:36.119189 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:36.119141 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jun 08 11:46:38.892193 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:38.892142 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Jun 08 11:46:39.498206 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:39.498164 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Jun 08 11:46:45.252787 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:45.252749 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Jun 08 11:46:46.119282 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:46.119240 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jun 08 11:46:48.892257 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:48.892209 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Jun 08 11:46:49.498048 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:49.498004 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Jun 08 11:46:55.253118 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:55.253037 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Jun 08 11:46:55.253476 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:55.253174 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:46:56.119463 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:56.119415 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jun 08 11:46:56.119671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:56.119553 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:46:58.892264 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:58.892223 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Jun 08 11:46:58.892736 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:58.892355 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:46:59.498215 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:59.498171 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Jun 08 11:46:59.498434 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:46:59.498322 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:47:01.573183 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.573119 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:47:01.723803 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.723768 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb323467-9b6d-4e87-a17f-354e9ab55e68-feast-data\") pod \"cb323467-9b6d-4e87-a17f-354e9ab55e68\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " Jun 08 11:47:01.723985 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.723842 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8q4z\" (UniqueName: \"kubernetes.io/projected/cb323467-9b6d-4e87-a17f-354e9ab55e68-kube-api-access-x8q4z\") pod \"cb323467-9b6d-4e87-a17f-354e9ab55e68\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " Jun 08 11:47:01.723985 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.723885 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-offline-tls\") pod \"cb323467-9b6d-4e87-a17f-354e9ab55e68\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " Jun 08 11:47:01.723985 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.723923 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-online-tls\") pod \"cb323467-9b6d-4e87-a17f-354e9ab55e68\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " Jun 08 11:47:01.724182 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.724109 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/cb323467-9b6d-4e87-a17f-354e9ab55e68-registry-tls\") pod \"cb323467-9b6d-4e87-a17f-354e9ab55e68\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " Jun 08 11:47:01.724182 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.724164 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-ui-tls\") pod \"cb323467-9b6d-4e87-a17f-354e9ab55e68\" (UID: \"cb323467-9b6d-4e87-a17f-354e9ab55e68\") " Jun 08 11:47:01.724489 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.724449 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb323467-9b6d-4e87-a17f-354e9ab55e68-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cb323467-9b6d-4e87-a17f-354e9ab55e68" (UID: "cb323467-9b6d-4e87-a17f-354e9ab55e68"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:47:01.724642 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.724494 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb323467-9b6d-4e87-a17f-354e9ab55e68-feast-data" (OuterVolumeSpecName: "feast-data") pod "cb323467-9b6d-4e87-a17f-354e9ab55e68" (UID: "cb323467-9b6d-4e87-a17f-354e9ab55e68"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:47:01.726347 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.726325 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb323467-9b6d-4e87-a17f-354e9ab55e68-kube-api-access-x8q4z" (OuterVolumeSpecName: "kube-api-access-x8q4z") pod "cb323467-9b6d-4e87-a17f-354e9ab55e68" (UID: "cb323467-9b6d-4e87-a17f-354e9ab55e68"). InnerVolumeSpecName "kube-api-access-x8q4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:47:01.726426 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.726373 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "cb323467-9b6d-4e87-a17f-354e9ab55e68" (UID: "cb323467-9b6d-4e87-a17f-354e9ab55e68"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:47:01.726666 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.726649 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "cb323467-9b6d-4e87-a17f-354e9ab55e68" (UID: "cb323467-9b6d-4e87-a17f-354e9ab55e68"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:47:01.726719 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.726673 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-online-tls" (OuterVolumeSpecName: "online-tls") pod "cb323467-9b6d-4e87-a17f-354e9ab55e68" (UID: "cb323467-9b6d-4e87-a17f-354e9ab55e68"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:47:01.824909 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.824871 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb323467-9b6d-4e87-a17f-354e9ab55e68-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:01.824909 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.824900 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8q4z\" (UniqueName: \"kubernetes.io/projected/cb323467-9b6d-4e87-a17f-354e9ab55e68-kube-api-access-x8q4z\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:01.824909 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.824913 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-offline-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:01.825145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.824924 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:01.825145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.824936 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/cb323467-9b6d-4e87-a17f-354e9ab55e68-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:01.825145 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.824946 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cb323467-9b6d-4e87-a17f-354e9ab55e68-ui-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:01.894800 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.894768 2569 generic.go:356] "Generic (PLEG): container finished" podID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerID="aa9558bc5eeded39697997acbd5fc182f33a1f8b375487eb51cc05617458f395" exitCode=137 Jun 08 11:47:01.894800 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.894792 2569 generic.go:356] "Generic (PLEG): container finished" podID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerID="0f6d06e90d42e472ec0e865bbe20d48e63901a8d9d4fae15f12ff126055652dc" exitCode=137 Jun 08 11:47:01.895029 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.894846 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerDied","Data":"aa9558bc5eeded39697997acbd5fc182f33a1f8b375487eb51cc05617458f395"} Jun 08 11:47:01.895029 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.894894 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerDied","Data":"0f6d06e90d42e472ec0e865bbe20d48e63901a8d9d4fae15f12ff126055652dc"} Jun 08 11:47:01.897142 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.897117 2569 generic.go:356] "Generic (PLEG): container finished" podID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerID="b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041" exitCode=137 Jun 08 11:47:01.897251 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.897150 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerDied","Data":"b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041"} Jun 08 11:47:01.897251 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.897174 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" event={"ID":"cb323467-9b6d-4e87-a17f-354e9ab55e68","Type":"ContainerDied","Data":"64a103029787e9da8204ced666d1de4288df01104c81847ee3886f7bae98cfd6"} Jun 08 11:47:01.897251 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.897184 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw" Jun 08 11:47:01.897251 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.897200 2569 scope.go:117] "RemoveContainer" containerID="fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e" Jun 08 11:47:01.907380 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.907361 2569 scope.go:117] "RemoveContainer" containerID="b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041" Jun 08 11:47:01.914898 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.914881 2569 scope.go:117] "RemoveContainer" containerID="a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5" Jun 08 11:47:01.922454 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.922438 2569 scope.go:117] "RemoveContainer" containerID="96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43" Jun 08 11:47:01.929944 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.929926 2569 scope.go:117] "RemoveContainer" containerID="d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2" Jun 08 11:47:01.935769 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.935743 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw"] Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.942234 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-qsxtw"] Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.942829 2569 scope.go:117] "RemoveContainer" containerID="fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:47:01.943264 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e\": container with ID starting with fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e not found: ID does not exist" containerID="fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.943301 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e"} err="failed to get container status \"fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e\": rpc error: code = NotFound desc = could not find container \"fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e\": container with ID starting with fea99ab66d3d73c6c57e4a18db1ac7cba18b91d5aa09727032bd79cc0bc8df0e not found: ID does not exist" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.943328 2569 scope.go:117] "RemoveContainer" containerID="b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:47:01.943647 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041\": container with ID starting with b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041 not found: ID does not exist" containerID="b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.943676 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041"} err="failed to get container status \"b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041\": rpc error: code = NotFound desc = could not find container \"b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041\": container with ID starting with b0ca8c5ead911540e0ac1e6d9ccbf29a089e351f1abbf05beba545891f458041 not found: ID does not exist" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.943696 2569 scope.go:117] "RemoveContainer" containerID="a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:47:01.943969 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5\": container with ID starting with a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5 not found: ID does not exist" containerID="a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.944003 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5"} err="failed to get container status \"a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5\": rpc error: code = NotFound desc = could not find container \"a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5\": container with ID starting with a44541e23046ed571eb1630cbd4aefa442ea6e4ea2b2e175212929d575a0a5c5 not found: ID does not exist" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.944025 2569 scope.go:117] "RemoveContainer" containerID="96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:47:01.944305 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43\": container with ID starting with 96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43 not found: ID does not exist" containerID="96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.944337 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43"} err="failed to get container status \"96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43\": rpc error: code = NotFound desc = could not find container \"96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43\": container with ID starting with 96b0a959961ca902be5bc56156062188dcf18f226f1ec1ab5e938ac750da7a43 not found: ID does not exist" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.944359 2569 scope.go:117] "RemoveContainer" containerID="d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:47:01.944770 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2\": container with ID starting with d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2 not found: ID does not exist" containerID="d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2" Jun 08 11:47:01.945034 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:01.944798 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2"} err="failed to get container status \"d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2\": rpc error: code = NotFound desc = could not find container \"d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2\": container with ID starting with d264ae71f351d76af9f9da987d4cee980c2033fe284e83565aa2a0193a1b83a2 not found: ID does not exist" Jun 08 11:47:02.191757 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.191728 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:47:02.227833 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.227804 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-ui-tls\") pod \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " Jun 08 11:47:02.228006 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.227848 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-offline-tls\") pod \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " Jun 08 11:47:02.228006 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.227983 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h52vd\" (UniqueName: \"kubernetes.io/projected/54fc71dc-cb96-450b-ba97-2b3a87fe0550-kube-api-access-h52vd\") pod \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " Jun 08 11:47:02.228125 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.228019 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/54fc71dc-cb96-450b-ba97-2b3a87fe0550-feast-data\") pod \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " Jun 08 11:47:02.228125 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.228058 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-online-tls\") pod \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " Jun 08 11:47:02.228125 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.228090 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-registry-tls\") pod \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\" (UID: \"54fc71dc-cb96-450b-ba97-2b3a87fe0550\") " Jun 08 11:47:02.228648 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.228596 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fc71dc-cb96-450b-ba97-2b3a87fe0550-feast-data" (OuterVolumeSpecName: "feast-data") pod "54fc71dc-cb96-450b-ba97-2b3a87fe0550" (UID: "54fc71dc-cb96-450b-ba97-2b3a87fe0550"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:47:02.230920 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.230817 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "54fc71dc-cb96-450b-ba97-2b3a87fe0550" (UID: "54fc71dc-cb96-450b-ba97-2b3a87fe0550"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:47:02.231072 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.231048 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "54fc71dc-cb96-450b-ba97-2b3a87fe0550" (UID: "54fc71dc-cb96-450b-ba97-2b3a87fe0550"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:47:02.231072 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.231051 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fc71dc-cb96-450b-ba97-2b3a87fe0550-kube-api-access-h52vd" (OuterVolumeSpecName: "kube-api-access-h52vd") pod "54fc71dc-cb96-450b-ba97-2b3a87fe0550" (UID: "54fc71dc-cb96-450b-ba97-2b3a87fe0550"). InnerVolumeSpecName "kube-api-access-h52vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:47:02.231219 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.231128 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "54fc71dc-cb96-450b-ba97-2b3a87fe0550" (UID: "54fc71dc-cb96-450b-ba97-2b3a87fe0550"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:47:02.231219 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.231191 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-online-tls" (OuterVolumeSpecName: "online-tls") pod "54fc71dc-cb96-450b-ba97-2b3a87fe0550" (UID: "54fc71dc-cb96-450b-ba97-2b3a87fe0550"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:47:02.328902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.328808 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h52vd\" (UniqueName: \"kubernetes.io/projected/54fc71dc-cb96-450b-ba97-2b3a87fe0550-kube-api-access-h52vd\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:02.328902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.328842 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/54fc71dc-cb96-450b-ba97-2b3a87fe0550-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:02.328902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.328852 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:02.328902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.328860 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:02.328902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.328867 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-ui-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:02.328902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.328874 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/54fc71dc-cb96-450b-ba97-2b3a87fe0550-offline-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:47:02.903803 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.903769 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" event={"ID":"54fc71dc-cb96-450b-ba97-2b3a87fe0550","Type":"ContainerDied","Data":"d05e9d1121e85cba1e18e5be3a1738f97c9d276efc1f94e4e67f15b151fffeb5"} Jun 08 11:47:02.904245 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.903825 2569 scope.go:117] "RemoveContainer" containerID="3511430e79f989fd9d8f0418e9b512872a8cdb193f2f8e86f95666a90a7350f8" Jun 08 11:47:02.904245 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.903773 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4" Jun 08 11:47:02.913690 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.913668 2569 scope.go:117] "RemoveContainer" containerID="aa9558bc5eeded39697997acbd5fc182f33a1f8b375487eb51cc05617458f395" Jun 08 11:47:02.921546 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.921527 2569 scope.go:117] "RemoveContainer" containerID="a9742f8e5f3bae92f73a73c2af83c289d0d971cdbbb6e394f94727eaa5ca6934" Jun 08 11:47:02.929469 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.929449 2569 scope.go:117] "RemoveContainer" containerID="0f6d06e90d42e472ec0e865bbe20d48e63901a8d9d4fae15f12ff126055652dc" Jun 08 11:47:02.930518 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.930498 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4"] Jun 08 11:47:02.937321 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.937305 2569 scope.go:117] "RemoveContainer" containerID="111082447b371602b3139c006fc74491fd09d1e0b0d2019a1e38b568757ca65f" Jun 08 11:47:02.942720 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.942695 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-xhdg4"] Jun 08 11:47:02.945699 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.945681 2569 scope.go:117] "RemoveContainer" containerID="41fcc99107750b0c69f9f41fc7d5d594571c39cfe26fe3a6be1e1616e0a4c3d2" Jun 08 11:47:02.953649 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.953604 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" path="/var/lib/kubelet/pods/54fc71dc-cb96-450b-ba97-2b3a87fe0550/volumes" Jun 08 11:47:02.954256 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:02.954242 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" path="/var/lib/kubelet/pods/cb323467-9b6d-4e87-a17f-354e9ab55e68/volumes" Jun 08 11:47:43.560150 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560110 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5"] Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560432 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="registry" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560442 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="registry" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560450 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="feast-init" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560456 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="feast-init" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560463 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="feast-apply" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560469 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="feast-apply" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560477 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="ui" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560482 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="ui" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560491 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="offline" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560496 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="offline" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560502 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="online" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560507 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="online" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560516 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="feast-init" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560522 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="feast-init" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560528 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="online" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560533 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="online" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560539 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="ui" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560544 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="ui" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560551 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="feast-apply" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560555 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="feast-apply" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560564 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="offline" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560568 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="offline" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560638 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="online" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560650 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="offline" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560657 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="ui" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560664 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb323467-9b6d-4e87-a17f-354e9ab55e68" containerName="ui" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560670 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="online" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560677 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="registry" Jun 08 11:47:43.560671 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.560682 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="54fc71dc-cb96-450b-ba97-2b3a87fe0550" containerName="offline" Jun 08 11:47:43.564171 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.564153 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.566818 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.566791 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 08 11:47:43.566940 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.566844 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 08 11:47:43.566940 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.566910 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 08 11:47:43.567044 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.566956 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 08 11:47:43.567044 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.567004 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 08 11:47:43.567143 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.567111 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 08 11:47:43.567188 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.567180 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-nxntc\"" Jun 08 11:47:43.577343 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.577321 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5"] Jun 08 11:47:43.702134 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.702098 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.702134 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.702133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.702370 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.702170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvgj\" (UniqueName: \"kubernetes.io/projected/5d2f429e-7417-423e-bb7d-965fc9913afb-kube-api-access-sgvgj\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.702370 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.702248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.702370 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.702293 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.702370 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.702323 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5d2f429e-7417-423e-bb7d-965fc9913afb-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.802970 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.802930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.802970 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.802976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5d2f429e-7417-423e-bb7d-965fc9913afb-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.803229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.803026 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.803229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.803044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.803229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.803076 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvgj\" (UniqueName: \"kubernetes.io/projected/5d2f429e-7417-423e-bb7d-965fc9913afb-kube-api-access-sgvgj\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.803229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.803110 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.803229 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:47:43.803200 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 08 11:47:43.803504 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:47:43.803292 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls podName:5d2f429e-7417-423e-bb7d-965fc9913afb nodeName:}" failed. No retries permitted until 2026-06-08 11:47:44.303270716 +0000 UTC m=+1653.787669560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls") pod "feast-simple-feast-setup-75c64c888b-kswt5" (UID: "5d2f429e-7417-423e-bb7d-965fc9913afb") : secret "feast-simple-feast-setup-ui-tls" not found Jun 08 11:47:43.803504 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.803468 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5d2f429e-7417-423e-bb7d-965fc9913afb-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.805580 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.805551 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.805714 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.805633 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.805714 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.805681 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:43.814271 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:43.814213 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvgj\" (UniqueName: \"kubernetes.io/projected/5d2f429e-7417-423e-bb7d-965fc9913afb-kube-api-access-sgvgj\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:44.307867 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:44.307829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:44.308063 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:47:44.307979 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 08 11:47:44.308063 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:47:44.308040 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls podName:5d2f429e-7417-423e-bb7d-965fc9913afb nodeName:}" failed. No retries permitted until 2026-06-08 11:47:45.308023162 +0000 UTC m=+1654.792422004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls") pod "feast-simple-feast-setup-75c64c888b-kswt5" (UID: "5d2f429e-7417-423e-bb7d-965fc9913afb") : secret "feast-simple-feast-setup-ui-tls" not found Jun 08 11:47:45.317699 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:45.317654 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:45.320213 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:45.320192 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-kswt5\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:45.383463 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:45.383421 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:47:45.536404 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:45.536377 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5"] Jun 08 11:47:45.537806 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:47:45.537778 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d2f429e_7417_423e_bb7d_965fc9913afb.slice/crio-2ff659c2bda5be3f9c30fa0aa6012075cc35cfda5adaf3f6c3764934ba8aa50e WatchSource:0}: Error finding container 2ff659c2bda5be3f9c30fa0aa6012075cc35cfda5adaf3f6c3764934ba8aa50e: Status 404 returned error can't find the container with id 2ff659c2bda5be3f9c30fa0aa6012075cc35cfda5adaf3f6c3764934ba8aa50e Jun 08 11:47:46.077553 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:46.077515 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerStarted","Data":"f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3"} Jun 08 11:47:46.077764 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:46.077568 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerStarted","Data":"2ff659c2bda5be3f9c30fa0aa6012075cc35cfda5adaf3f6c3764934ba8aa50e"} Jun 08 11:47:51.099454 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:51.099418 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerID="f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3" exitCode=0 Jun 08 11:47:51.099454 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:51.099458 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerDied","Data":"f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3"} Jun 08 11:47:52.105293 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:52.105247 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerStarted","Data":"71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4"} Jun 08 11:47:57.129073 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:57.129037 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerID="71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4" exitCode=0 Jun 08 11:47:57.129485 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:57.129090 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerDied","Data":"71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4"} Jun 08 11:47:58.139967 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:58.139919 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerStarted","Data":"6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1"} Jun 08 11:47:58.140419 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:58.139993 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerStarted","Data":"27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65"} Jun 08 11:47:58.140419 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:58.140014 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerStarted","Data":"6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c"} Jun 08 11:47:58.140419 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:58.140034 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerStarted","Data":"27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c"} Jun 08 11:47:58.164180 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:47:58.164110 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podStartSLOduration=15.164087469 podStartE2EDuration="15.164087469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:47:58.162520471 +0000 UTC m=+1667.646919359" watchObservedRunningTime="2026-06-08 11:47:58.164087469 +0000 UTC m=+1667.648486334" Jun 08 11:48:00.383881 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:00.383847 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:00.384445 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:00.384422 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:00.384594 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:00.384453 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:00.384594 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:00.384471 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:00.386056 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:00.386024 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused" Jun 08 11:48:00.386171 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:00.386126 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused" Jun 08 11:48:00.386263 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:00.386034 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Jun 08 11:48:00.386324 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:00.386095 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Jun 08 11:48:03.385008 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:03.384973 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:03.385543 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:03.385519 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:03.385743 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:03.385728 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:03.385790 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:03.385761 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:03.385836 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:03.385825 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:03.389697 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:03.389677 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:04.173814 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:04.173775 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:04.174022 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:04.173834 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:04.174022 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:04.173854 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:04.174297 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:04.174274 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:04.174376 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:04.174337 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:48:04.176832 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:48:04.176813 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:49:05.662301 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:05.662265 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5"] Jun 08 11:49:05.663211 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:05.663152 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="registry" containerID="cri-o://27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c" gracePeriod=30 Jun 08 11:49:05.663442 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:05.663213 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="offline" containerID="cri-o://27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65" gracePeriod=30 Jun 08 11:49:05.663593 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:05.663232 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="online" containerID="cri-o://6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c" gracePeriod=30 Jun 08 11:49:05.663728 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:05.663295 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="ui" containerID="cri-o://6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1" gracePeriod=30 Jun 08 11:49:05.943251 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:05.943168 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb"] Jun 08 11:49:05.947155 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:05.947137 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:05.949724 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:05.949695 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-v5xbb\"" Jun 08 11:49:05.960012 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:05.959977 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb"] Jun 08 11:49:06.030914 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.030874 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fdf7a6b-7a68-404c-8ec7-643212c1095e-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.031134 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.031024 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.031134 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.031066 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxrw\" (UniqueName: \"kubernetes.io/projected/4fdf7a6b-7a68-404c-8ec7-643212c1095e-kube-api-access-kcxrw\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.031233 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.031202 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.031286 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.031268 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.031334 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.031315 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.132542 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.132494 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.132542 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.132545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxrw\" (UniqueName: \"kubernetes.io/projected/4fdf7a6b-7a68-404c-8ec7-643212c1095e-kube-api-access-kcxrw\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.132776 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.132623 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.132776 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.132658 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.132776 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.132693 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.132776 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.132743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fdf7a6b-7a68-404c-8ec7-643212c1095e-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.133211 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.133189 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fdf7a6b-7a68-404c-8ec7-643212c1095e-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.133333 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:49:06.133318 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 08 11:49:06.133396 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:49:06.133387 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-registry-tls podName:4fdf7a6b-7a68-404c-8ec7-643212c1095e nodeName:}" failed. No retries permitted until 2026-06-08 11:49:06.633365938 +0000 UTC m=+1736.117764795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-registry-tls") pod "feast-simple-feast-setup-75c64c888b-s76nb" (UID: "4fdf7a6b-7a68-404c-8ec7-643212c1095e") : secret "feast-simple-feast-setup-registry-tls" not found Jun 08 11:49:06.135856 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.135825 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.136463 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.136437 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.138456 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.138430 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.144085 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.144055 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxrw\" (UniqueName: \"kubernetes.io/projected/4fdf7a6b-7a68-404c-8ec7-643212c1095e-kube-api-access-kcxrw\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.433595 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.433562 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerID="6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1" exitCode=0 Jun 08 11:49:06.433595 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.433589 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerID="6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c" exitCode=0 Jun 08 11:49:06.433814 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.433640 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerDied","Data":"6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1"} Jun 08 11:49:06.433814 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.433680 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerDied","Data":"6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c"} Jun 08 11:49:06.637539 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.637506 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.640019 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.639998 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-s76nb\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:06.856715 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:06.856663 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:07.000660 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:07.000636 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb"] Jun 08 11:49:07.003274 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:49:07.003243 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fdf7a6b_7a68_404c_8ec7_643212c1095e.slice/crio-a7ff5949e6b6fecad4520012b10c951f3d3a693e731a691046070c154a60a96f WatchSource:0}: Error finding container a7ff5949e6b6fecad4520012b10c951f3d3a693e731a691046070c154a60a96f: Status 404 returned error can't find the container with id a7ff5949e6b6fecad4520012b10c951f3d3a693e731a691046070c154a60a96f Jun 08 11:49:07.439874 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:07.439829 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerStarted","Data":"3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d"} Jun 08 11:49:07.440064 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:07.439893 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerStarted","Data":"a7ff5949e6b6fecad4520012b10c951f3d3a693e731a691046070c154a60a96f"} Jun 08 11:49:12.465828 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:12.465794 2569 generic.go:356] "Generic (PLEG): container finished" podID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerID="3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d" exitCode=0 Jun 08 11:49:12.466246 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:12.465874 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerDied","Data":"3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d"} Jun 08 11:49:13.473400 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:13.473365 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerStarted","Data":"d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b"} Jun 08 11:49:14.174357 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:14.174313 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Jun 08 11:49:14.174571 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:14.174320 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Jun 08 11:49:18.498782 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:18.498742 2569 generic.go:356] "Generic (PLEG): container finished" podID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerID="d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b" exitCode=0 Jun 08 11:49:18.499151 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:18.498798 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerDied","Data":"d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b"} Jun 08 11:49:19.510053 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:19.510013 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerStarted","Data":"fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b"} Jun 08 11:49:19.510662 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:19.510078 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerStarted","Data":"d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207"} Jun 08 11:49:19.510662 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:19.510100 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerStarted","Data":"804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9"} Jun 08 11:49:19.510662 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:19.510117 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerStarted","Data":"edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78"} Jun 08 11:49:19.541167 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:19.541101 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podStartSLOduration=14.541078403 podStartE2EDuration="14.541078403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:49:19.537549059 +0000 UTC m=+1749.021947923" watchObservedRunningTime="2026-06-08 11:49:19.541078403 +0000 UTC m=+1749.025477267" Jun 08 11:49:21.858076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:21.857805 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:21.858076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:21.857884 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:21.858076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:21.857906 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:21.858076 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:21.857997 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:21.860502 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:21.860366 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Jun 08 11:49:21.860502 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:21.860404 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Jun 08 11:49:21.860502 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:21.860417 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jun 08 11:49:21.860502 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:21.860496 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.36:8816: connect: connection refused" Jun 08 11:49:24.174766 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.174715 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Jun 08 11:49:24.175304 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.174727 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Jun 08 11:49:24.858163 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.858133 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:24.858352 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.858228 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:24.858472 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.858455 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:24.858820 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.858794 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:24.858963 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.858833 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:24.858963 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.858851 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:24.859083 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.859029 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:24.859157 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.859136 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:24.859269 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.859254 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:24.862518 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:24.862495 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:25.545223 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:25.545192 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:25.548324 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:25.548303 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:49:34.173981 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:34.173929 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Jun 08 11:49:34.174401 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:34.173922 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Jun 08 11:49:34.174401 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:34.174077 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:49:34.174401 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:34.174150 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:49:36.317101 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.317072 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:49:36.429501 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.429465 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-registry-tls\") pod \"5d2f429e-7417-423e-bb7d-965fc9913afb\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " Jun 08 11:49:36.429501 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.429509 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5d2f429e-7417-423e-bb7d-965fc9913afb-feast-data\") pod \"5d2f429e-7417-423e-bb7d-965fc9913afb\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " Jun 08 11:49:36.429765 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.429648 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgvgj\" (UniqueName: \"kubernetes.io/projected/5d2f429e-7417-423e-bb7d-965fc9913afb-kube-api-access-sgvgj\") pod \"5d2f429e-7417-423e-bb7d-965fc9913afb\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " Jun 08 11:49:36.429765 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.429679 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls\") pod \"5d2f429e-7417-423e-bb7d-965fc9913afb\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " Jun 08 11:49:36.429765 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.429697 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-online-tls\") pod \"5d2f429e-7417-423e-bb7d-965fc9913afb\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " Jun 08 11:49:36.429765 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.429738 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-offline-tls\") pod \"5d2f429e-7417-423e-bb7d-965fc9913afb\" (UID: \"5d2f429e-7417-423e-bb7d-965fc9913afb\") " Jun 08 11:49:36.430083 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.430021 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2f429e-7417-423e-bb7d-965fc9913afb-feast-data" (OuterVolumeSpecName: "feast-data") pod "5d2f429e-7417-423e-bb7d-965fc9913afb" (UID: "5d2f429e-7417-423e-bb7d-965fc9913afb"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:49:36.432029 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.432003 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5d2f429e-7417-423e-bb7d-965fc9913afb" (UID: "5d2f429e-7417-423e-bb7d-965fc9913afb"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:49:36.432132 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.432081 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2f429e-7417-423e-bb7d-965fc9913afb-kube-api-access-sgvgj" (OuterVolumeSpecName: "kube-api-access-sgvgj") pod "5d2f429e-7417-423e-bb7d-965fc9913afb" (UID: "5d2f429e-7417-423e-bb7d-965fc9913afb"). InnerVolumeSpecName "kube-api-access-sgvgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:49:36.432132 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.432128 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5d2f429e-7417-423e-bb7d-965fc9913afb" (UID: "5d2f429e-7417-423e-bb7d-965fc9913afb"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:49:36.432243 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.432223 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5d2f429e-7417-423e-bb7d-965fc9913afb" (UID: "5d2f429e-7417-423e-bb7d-965fc9913afb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:49:36.432281 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.432246 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-online-tls" (OuterVolumeSpecName: "online-tls") pod "5d2f429e-7417-423e-bb7d-965fc9913afb" (UID: "5d2f429e-7417-423e-bb7d-965fc9913afb"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:49:36.530831 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.530793 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgvgj\" (UniqueName: \"kubernetes.io/projected/5d2f429e-7417-423e-bb7d-965fc9913afb-kube-api-access-sgvgj\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:49:36.530831 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.530826 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-ui-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:49:36.530831 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.530838 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:49:36.531061 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.530847 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-offline-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:49:36.531061 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.530855 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5d2f429e-7417-423e-bb7d-965fc9913afb-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:49:36.531061 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.530862 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5d2f429e-7417-423e-bb7d-965fc9913afb-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:49:36.593840 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.593805 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerID="27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65" exitCode=137 Jun 08 11:49:36.593840 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.593829 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerID="27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c" exitCode=137 Jun 08 11:49:36.594072 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.593854 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerDied","Data":"27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65"} Jun 08 11:49:36.594072 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.593874 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" Jun 08 11:49:36.594072 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.593900 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerDied","Data":"27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c"} Jun 08 11:49:36.594072 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.593919 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5" event={"ID":"5d2f429e-7417-423e-bb7d-965fc9913afb","Type":"ContainerDied","Data":"2ff659c2bda5be3f9c30fa0aa6012075cc35cfda5adaf3f6c3764934ba8aa50e"} Jun 08 11:49:36.594072 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.593942 2569 scope.go:117] "RemoveContainer" containerID="6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1" Jun 08 11:49:36.603123 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.603107 2569 scope.go:117] "RemoveContainer" containerID="27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65" Jun 08 11:49:36.611170 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.611154 2569 scope.go:117] "RemoveContainer" containerID="6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c" Jun 08 11:49:36.615872 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.615851 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5"] Jun 08 11:49:36.618938 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.618909 2569 scope.go:117] "RemoveContainer" containerID="27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c" Jun 08 11:49:36.620282 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.620263 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-kswt5"] Jun 08 11:49:36.626495 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.626477 2569 scope.go:117] "RemoveContainer" containerID="71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4" Jun 08 11:49:36.633884 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.633868 2569 scope.go:117] "RemoveContainer" containerID="f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3" Jun 08 11:49:36.641166 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.641142 2569 scope.go:117] "RemoveContainer" containerID="6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1" Jun 08 11:49:36.641421 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:49:36.641401 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1\": container with ID starting with 6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1 not found: ID does not exist" containerID="6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1" Jun 08 11:49:36.641478 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.641436 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1"} err="failed to get container status \"6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1\": rpc error: code = NotFound desc = could not find container \"6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1\": container with ID starting with 6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1 not found: ID does not exist" Jun 08 11:49:36.641478 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.641456 2569 scope.go:117] "RemoveContainer" containerID="27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65" Jun 08 11:49:36.641710 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:49:36.641688 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65\": container with ID starting with 27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65 not found: ID does not exist" containerID="27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65" Jun 08 11:49:36.641771 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.641720 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65"} err="failed to get container status \"27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65\": rpc error: code = NotFound desc = could not find container \"27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65\": container with ID starting with 27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65 not found: ID does not exist" Jun 08 11:49:36.641771 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.641743 2569 scope.go:117] "RemoveContainer" containerID="6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c" Jun 08 11:49:36.641990 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:49:36.641970 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c\": container with ID starting with 6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c not found: ID does not exist" containerID="6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c" Jun 08 11:49:36.642060 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.641999 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c"} err="failed to get container status \"6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c\": rpc error: code = NotFound desc = could not find container \"6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c\": container with ID starting with 6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c not found: ID does not exist" Jun 08 11:49:36.642060 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.642021 2569 scope.go:117] "RemoveContainer" containerID="27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c" Jun 08 11:49:36.642253 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:49:36.642237 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c\": container with ID starting with 27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c not found: ID does not exist" containerID="27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c" Jun 08 11:49:36.642314 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.642258 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c"} err="failed to get container status \"27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c\": rpc error: code = NotFound desc = could not find container \"27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c\": container with ID starting with 27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c not found: ID does not exist" Jun 08 11:49:36.642314 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.642276 2569 scope.go:117] "RemoveContainer" containerID="71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4" Jun 08 11:49:36.642507 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:49:36.642490 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4\": container with ID starting with 71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4 not found: ID does not exist" containerID="71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4" Jun 08 11:49:36.642571 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.642513 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4"} err="failed to get container status \"71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4\": rpc error: code = NotFound desc = could not find container \"71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4\": container with ID starting with 71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4 not found: ID does not exist" Jun 08 11:49:36.642571 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.642535 2569 scope.go:117] "RemoveContainer" containerID="f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3" Jun 08 11:49:36.642773 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:49:36.642756 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3\": container with ID starting with f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3 not found: ID does not exist" containerID="f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3" Jun 08 11:49:36.642841 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.642780 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3"} err="failed to get container status \"f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3\": rpc error: code = NotFound desc = could not find container \"f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3\": container with ID starting with f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3 not found: ID does not exist" Jun 08 11:49:36.642841 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.642798 2569 scope.go:117] "RemoveContainer" containerID="6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1" Jun 08 11:49:36.643048 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.643030 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1"} err="failed to get container status \"6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1\": rpc error: code = NotFound desc = could not find container \"6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1\": container with ID starting with 6d1058fa3227368206291d283dd1148e8e2ff89c0879896e0b0800216c5316b1 not found: ID does not exist" Jun 08 11:49:36.643098 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.643050 2569 scope.go:117] "RemoveContainer" containerID="27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65" Jun 08 11:49:36.643263 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.643246 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65"} err="failed to get container status \"27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65\": rpc error: code = NotFound desc = could not find container \"27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65\": container with ID starting with 27007f0223ed351f9af4f7b61d729bc1028738288bfe9eeb935022c49a486a65 not found: ID does not exist" Jun 08 11:49:36.643308 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.643273 2569 scope.go:117] "RemoveContainer" containerID="6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c" Jun 08 11:49:36.643477 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.643459 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c"} err="failed to get container status \"6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c\": rpc error: code = NotFound desc = could not find container \"6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c\": container with ID starting with 6297881419f5a1e6da7c0a9799af7c24bb80608068c9849d7354e9fedcb9457c not found: ID does not exist" Jun 08 11:49:36.643546 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.643480 2569 scope.go:117] "RemoveContainer" containerID="27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c" Jun 08 11:49:36.643743 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.643726 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c"} err="failed to get container status \"27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c\": rpc error: code = NotFound desc = could not find container \"27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c\": container with ID starting with 27476c7d91720e34a9b70b7c25b3caa95ea65c0164ebbc7ab3a32bc3a968e66c not found: ID does not exist" Jun 08 11:49:36.643816 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.643744 2569 scope.go:117] "RemoveContainer" containerID="71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4" Jun 08 11:49:36.643948 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.643928 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4"} err="failed to get container status \"71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4\": rpc error: code = NotFound desc = could not find container \"71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4\": container with ID starting with 71a370c2b7ae4aaf874588f3752af6a1d70b163d298e4348bf3381b48e487eb4 not found: ID does not exist" Jun 08 11:49:36.643995 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.643949 2569 scope.go:117] "RemoveContainer" containerID="f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3" Jun 08 11:49:36.644158 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.644140 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3"} err="failed to get container status \"f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3\": rpc error: code = NotFound desc = could not find container \"f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3\": container with ID starting with f3b95ca8966bedec99f811bbd573c29ec86dadb8daf8a1f3d002b576bfd094c3 not found: ID does not exist" Jun 08 11:49:36.953668 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:49:36.953636 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" path="/var/lib/kubelet/pods/5d2f429e-7417-423e-bb7d-965fc9913afb/volumes" Jun 08 11:50:27.403690 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.403654 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6"] Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.403977 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="online" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.403988 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="online" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.403999 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="feast-init" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404005 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="feast-init" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404013 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="ui" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404018 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="ui" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404025 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="registry" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404031 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="registry" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404041 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="offline" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404046 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="offline" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404062 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="feast-apply" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404067 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="feast-apply" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404119 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="registry" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404126 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="online" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404134 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="ui" Jun 08 11:50:27.406102 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.404142 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2f429e-7417-423e-bb7d-965fc9913afb" containerName="offline" Jun 08 11:50:27.407780 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.407764 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.410065 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.410044 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 08 11:50:27.410177 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.410130 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 08 11:50:27.410228 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.410164 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 08 11:50:27.410337 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.410321 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-t76zt\"" Jun 08 11:50:27.411151 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.411136 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 08 11:50:27.411220 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.411212 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 08 11:50:27.411448 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.411435 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 08 11:50:27.427991 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.427959 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6"] Jun 08 11:50:27.563907 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.563872 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.564104 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.563920 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.564104 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.563982 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.564104 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.564022 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.564104 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.564042 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9l9q\" (UniqueName: \"kubernetes.io/projected/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-kube-api-access-m9l9q\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.564104 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.564076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.664962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.664879 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.664962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.664929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.664962 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.664958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.665229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.664992 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.665229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.665128 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.665229 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.665173 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9l9q\" (UniqueName: \"kubernetes.io/projected/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-kube-api-access-m9l9q\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.665396 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.665329 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.665762 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.665728 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.667707 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.667674 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.667830 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.667797 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.667830 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.667809 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.673290 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.673265 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9l9q\" (UniqueName: \"kubernetes.io/projected/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-kube-api-access-m9l9q\") pod \"feast-simple-feast-remote-setup-669bcf45fc-j6vv6\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.717481 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.717451 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:27.855563 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:27.855532 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6"] Jun 08 11:50:27.859070 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:50:27.859046 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode250596e_e1dc_4fb3_a66f_8e26a144d9a5.slice/crio-ea5ea50ea2b28923c693d7099fe6ac6e7df5bee3f850499251a1017b0369d877 WatchSource:0}: Error finding container ea5ea50ea2b28923c693d7099fe6ac6e7df5bee3f850499251a1017b0369d877: Status 404 returned error can't find the container with id ea5ea50ea2b28923c693d7099fe6ac6e7df5bee3f850499251a1017b0369d877 Jun 08 11:50:28.807756 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:28.807712 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerStarted","Data":"9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60"} Jun 08 11:50:28.808128 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:28.807768 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerStarted","Data":"ea5ea50ea2b28923c693d7099fe6ac6e7df5bee3f850499251a1017b0369d877"} Jun 08 11:50:32.826816 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:32.826782 2569 generic.go:356] "Generic (PLEG): container finished" podID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerID="9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60" exitCode=0 Jun 08 11:50:32.827236 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:32.826831 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerDied","Data":"9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60"} Jun 08 11:50:33.839357 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:33.839314 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerStarted","Data":"cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc"} Jun 08 11:50:38.863865 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:38.863831 2569 generic.go:356] "Generic (PLEG): container finished" podID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerID="cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc" exitCode=0 Jun 08 11:50:38.864271 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:38.863911 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerDied","Data":"cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc"} Jun 08 11:50:39.881551 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:39.881500 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerStarted","Data":"884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661"} Jun 08 11:50:39.882045 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:39.881564 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerStarted","Data":"7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5"} Jun 08 11:50:39.882045 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:39.881582 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerStarted","Data":"48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152"} Jun 08 11:50:39.905953 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:39.905894 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podStartSLOduration=12.905877273 podStartE2EDuration="12.905877273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:50:39.904148553 +0000 UTC m=+1829.388547429" watchObservedRunningTime="2026-06-08 11:50:39.905877273 +0000 UTC m=+1829.390276135" Jun 08 11:50:42.718170 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:42.718116 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:42.718170 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:42.718182 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:42.718781 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:42.718202 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:42.720309 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:42.720266 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused" Jun 08 11:50:42.720518 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:42.720312 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jun 08 11:50:42.720518 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:42.720318 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Jun 08 11:50:45.718761 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:45.718727 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:45.719208 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:45.718852 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:45.719284 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:45.719229 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:45.719284 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:45.719268 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:45.719632 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:45.719591 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:45.719751 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:45.719737 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:45.722669 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:45.722655 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:45.915393 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:45.915363 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:50:45.918287 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:50:45.918268 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:51:48.379653 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:48.379597 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6"] Jun 08 11:51:48.380140 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:48.379959 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="online" containerID="cri-o://48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152" gracePeriod=30 Jun 08 11:51:48.380140 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:48.380052 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="ui" containerID="cri-o://884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661" gracePeriod=30 Jun 08 11:51:48.380279 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:48.380062 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="offline" containerID="cri-o://7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5" gracePeriod=30 Jun 08 11:51:48.526233 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:48.526200 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb"] Jun 08 11:51:48.526772 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:48.526456 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="registry" containerID="cri-o://edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78" gracePeriod=30 Jun 08 11:51:48.526772 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:48.526527 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="offline" containerID="cri-o://d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207" gracePeriod=30 Jun 08 11:51:48.526772 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:48.526527 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="online" containerID="cri-o://804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9" gracePeriod=30 Jun 08 11:51:48.526772 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:48.526696 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="ui" containerID="cri-o://fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b" gracePeriod=30 Jun 08 11:51:49.186353 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:49.186313 2569 generic.go:356] "Generic (PLEG): container finished" podID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerID="884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661" exitCode=0 Jun 08 11:51:49.186353 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:49.186349 2569 generic.go:356] "Generic (PLEG): container finished" podID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerID="48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152" exitCode=0 Jun 08 11:51:49.186562 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:49.186391 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerDied","Data":"884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661"} Jun 08 11:51:49.186562 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:49.186441 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerDied","Data":"48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152"} Jun 08 11:51:49.189458 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:49.189432 2569 generic.go:356] "Generic (PLEG): container finished" podID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerID="fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b" exitCode=0 Jun 08 11:51:49.189583 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:49.189480 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerDied","Data":"fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b"} Jun 08 11:51:50.196424 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:50.196391 2569 generic.go:356] "Generic (PLEG): container finished" podID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerID="804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9" exitCode=0 Jun 08 11:51:50.196819 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:50.196472 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerDied","Data":"804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9"} Jun 08 11:51:54.859455 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:54.859404 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Jun 08 11:51:55.545931 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:55.545887 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jun 08 11:51:55.720338 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:55.720293 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Jun 08 11:51:55.916175 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:51:55.916071 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jun 08 11:52:04.859343 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:04.859295 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Jun 08 11:52:05.546339 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:05.546299 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jun 08 11:52:05.719902 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:05.719573 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Jun 08 11:52:05.916206 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:05.916108 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jun 08 11:52:14.859114 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:14.859063 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Jun 08 11:52:14.859536 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:14.859218 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:52:15.546279 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:15.546240 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jun 08 11:52:15.546493 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:15.546354 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:52:15.719680 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:15.719602 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Jun 08 11:52:15.719865 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:15.719765 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:52:15.916474 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:15.916373 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jun 08 11:52:15.916857 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:15.916523 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:52:18.650529 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.650455 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:52:18.758136 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.758103 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-online-tls\") pod \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " Jun 08 11:52:18.758287 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.758152 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-ui-tls\") pod \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " Jun 08 11:52:18.758287 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.758188 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-registry-tls\") pod \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " Jun 08 11:52:18.758287 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.758208 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-offline-tls\") pod \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " Jun 08 11:52:18.758287 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.758242 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9l9q\" (UniqueName: \"kubernetes.io/projected/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-kube-api-access-m9l9q\") pod \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " Jun 08 11:52:18.758287 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.758264 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-feast-data\") pod \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\" (UID: \"e250596e-e1dc-4fb3-a66f-8e26a144d9a5\") " Jun 08 11:52:18.758685 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.758647 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e250596e-e1dc-4fb3-a66f-8e26a144d9a5" (UID: "e250596e-e1dc-4fb3-a66f-8e26a144d9a5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:52:18.758856 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.758829 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-feast-data" (OuterVolumeSpecName: "feast-data") pod "e250596e-e1dc-4fb3-a66f-8e26a144d9a5" (UID: "e250596e-e1dc-4fb3-a66f-8e26a144d9a5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:52:18.760836 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.760811 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-online-tls" (OuterVolumeSpecName: "online-tls") pod "e250596e-e1dc-4fb3-a66f-8e26a144d9a5" (UID: "e250596e-e1dc-4fb3-a66f-8e26a144d9a5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:52:18.760927 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.760829 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e250596e-e1dc-4fb3-a66f-8e26a144d9a5" (UID: "e250596e-e1dc-4fb3-a66f-8e26a144d9a5"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:52:18.760927 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.760834 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-kube-api-access-m9l9q" (OuterVolumeSpecName: "kube-api-access-m9l9q") pod "e250596e-e1dc-4fb3-a66f-8e26a144d9a5" (UID: "e250596e-e1dc-4fb3-a66f-8e26a144d9a5"). InnerVolumeSpecName "kube-api-access-m9l9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:52:18.760927 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.760851 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e250596e-e1dc-4fb3-a66f-8e26a144d9a5" (UID: "e250596e-e1dc-4fb3-a66f-8e26a144d9a5"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:52:18.858996 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.858964 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:18.858996 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.858993 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-ui-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:18.859197 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.859006 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:18.859197 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.859017 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-offline-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:18.859197 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.859028 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9l9q\" (UniqueName: \"kubernetes.io/projected/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-kube-api-access-m9l9q\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:18.859197 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:18.859040 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e250596e-e1dc-4fb3-a66f-8e26a144d9a5-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:19.177756 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.177731 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:52:19.262492 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.262467 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-offline-tls\") pod \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " Jun 08 11:52:19.262676 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.262538 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-ui-tls\") pod \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " Jun 08 11:52:19.262676 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.262587 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcxrw\" (UniqueName: \"kubernetes.io/projected/4fdf7a6b-7a68-404c-8ec7-643212c1095e-kube-api-access-kcxrw\") pod \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " Jun 08 11:52:19.262676 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.262626 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fdf7a6b-7a68-404c-8ec7-643212c1095e-feast-data\") pod \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " Jun 08 11:52:19.262676 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.262646 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-online-tls\") pod \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " Jun 08 11:52:19.262913 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.262753 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-registry-tls\") pod \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\" (UID: \"4fdf7a6b-7a68-404c-8ec7-643212c1095e\") " Jun 08 11:52:19.263258 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.263152 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fdf7a6b-7a68-404c-8ec7-643212c1095e-feast-data" (OuterVolumeSpecName: "feast-data") pod "4fdf7a6b-7a68-404c-8ec7-643212c1095e" (UID: "4fdf7a6b-7a68-404c-8ec7-643212c1095e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:52:19.264921 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.264890 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-online-tls" (OuterVolumeSpecName: "online-tls") pod "4fdf7a6b-7a68-404c-8ec7-643212c1095e" (UID: "4fdf7a6b-7a68-404c-8ec7-643212c1095e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:52:19.264921 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.264911 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4fdf7a6b-7a68-404c-8ec7-643212c1095e" (UID: "4fdf7a6b-7a68-404c-8ec7-643212c1095e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:52:19.265099 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.264992 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "4fdf7a6b-7a68-404c-8ec7-643212c1095e" (UID: "4fdf7a6b-7a68-404c-8ec7-643212c1095e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:52:19.265173 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.265157 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fdf7a6b-7a68-404c-8ec7-643212c1095e-kube-api-access-kcxrw" (OuterVolumeSpecName: "kube-api-access-kcxrw") pod "4fdf7a6b-7a68-404c-8ec7-643212c1095e" (UID: "4fdf7a6b-7a68-404c-8ec7-643212c1095e"). InnerVolumeSpecName "kube-api-access-kcxrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:52:19.265330 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.265317 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4fdf7a6b-7a68-404c-8ec7-643212c1095e" (UID: "4fdf7a6b-7a68-404c-8ec7-643212c1095e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:52:19.320868 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.320835 2569 generic.go:356] "Generic (PLEG): container finished" podID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerID="7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5" exitCode=137 Jun 08 11:52:19.321025 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.320898 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" Jun 08 11:52:19.321025 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.320921 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerDied","Data":"7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5"} Jun 08 11:52:19.321025 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.320974 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6" event={"ID":"e250596e-e1dc-4fb3-a66f-8e26a144d9a5","Type":"ContainerDied","Data":"ea5ea50ea2b28923c693d7099fe6ac6e7df5bee3f850499251a1017b0369d877"} Jun 08 11:52:19.321025 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.320999 2569 scope.go:117] "RemoveContainer" containerID="884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661" Jun 08 11:52:19.324326 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.324297 2569 generic.go:356] "Generic (PLEG): container finished" podID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerID="d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207" exitCode=137 Jun 08 11:52:19.324326 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.324325 2569 generic.go:356] "Generic (PLEG): container finished" podID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerID="edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78" exitCode=137 Jun 08 11:52:19.324326 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.324325 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerDied","Data":"d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207"} Jun 08 11:52:19.324326 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.324364 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerDied","Data":"edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78"} Jun 08 11:52:19.324326 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.324375 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" Jun 08 11:52:19.324326 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.324388 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb" event={"ID":"4fdf7a6b-7a68-404c-8ec7-643212c1095e","Type":"ContainerDied","Data":"a7ff5949e6b6fecad4520012b10c951f3d3a693e731a691046070c154a60a96f"} Jun 08 11:52:19.332683 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.332662 2569 scope.go:117] "RemoveContainer" containerID="7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5" Jun 08 11:52:19.340745 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.340727 2569 scope.go:117] "RemoveContainer" containerID="48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152" Jun 08 11:52:19.344079 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.344055 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6"] Jun 08 11:52:19.348537 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.348521 2569 scope.go:117] "RemoveContainer" containerID="cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc" Jun 08 11:52:19.352289 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.352270 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-j6vv6"] Jun 08 11:52:19.356466 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.356451 2569 scope.go:117] "RemoveContainer" containerID="9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60" Jun 08 11:52:19.363676 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.363602 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcxrw\" (UniqueName: \"kubernetes.io/projected/4fdf7a6b-7a68-404c-8ec7-643212c1095e-kube-api-access-kcxrw\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:19.363740 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.363694 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fdf7a6b-7a68-404c-8ec7-643212c1095e-feast-data\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:19.363740 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.363719 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-online-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:19.363740 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.363731 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-registry-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:19.363740 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.363697 2569 scope.go:117] "RemoveContainer" containerID="884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661" Jun 08 11:52:19.363740 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.363738 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-offline-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:19.363920 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.363759 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4fdf7a6b-7a68-404c-8ec7-643212c1095e-ui-tls\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:52:19.364023 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:52:19.364007 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661\": container with ID starting with 884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661 not found: ID does not exist" containerID="884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661" Jun 08 11:52:19.364068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.364040 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661"} err="failed to get container status \"884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661\": rpc error: code = NotFound desc = could not find container \"884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661\": container with ID starting with 884e9fadc7c81450c3011c185c053b5bef91960e3d531f5588408ea56a204661 not found: ID does not exist" Jun 08 11:52:19.364068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.364056 2569 scope.go:117] "RemoveContainer" containerID="7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5" Jun 08 11:52:19.364285 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:52:19.364269 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5\": container with ID starting with 7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5 not found: ID does not exist" containerID="7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5" Jun 08 11:52:19.364326 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.364300 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5"} err="failed to get container status \"7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5\": rpc error: code = NotFound desc = could not find container \"7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5\": container with ID starting with 7931e026f173736714d78c7085a4f26f4d59b2178d87492b23087c5107547bf5 not found: ID does not exist" Jun 08 11:52:19.364326 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.364320 2569 scope.go:117] "RemoveContainer" containerID="48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152" Jun 08 11:52:19.364531 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:52:19.364517 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152\": container with ID starting with 48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152 not found: ID does not exist" containerID="48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152" Jun 08 11:52:19.364570 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.364538 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152"} err="failed to get container status \"48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152\": rpc error: code = NotFound desc = could not find container \"48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152\": container with ID starting with 48b6c71fc6271bee261c465e565ddb6e5c41efe1a4d1500fae944e5ee4c45152 not found: ID does not exist" Jun 08 11:52:19.364570 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.364553 2569 scope.go:117] "RemoveContainer" containerID="cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc" Jun 08 11:52:19.364801 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:52:19.364785 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc\": container with ID starting with cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc not found: ID does not exist" containerID="cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc" Jun 08 11:52:19.364841 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.364805 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc"} err="failed to get container status \"cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc\": rpc error: code = NotFound desc = could not find container \"cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc\": container with ID starting with cff737b83a0e6db84735db905aa6b9456922694d09793f7a46313e61ca3efcdc not found: ID does not exist" Jun 08 11:52:19.364841 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.364818 2569 scope.go:117] "RemoveContainer" containerID="9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60" Jun 08 11:52:19.365024 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:52:19.365007 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60\": container with ID starting with 9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60 not found: ID does not exist" containerID="9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60" Jun 08 11:52:19.365068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.365029 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60"} err="failed to get container status \"9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60\": rpc error: code = NotFound desc = could not find container \"9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60\": container with ID starting with 9425d5aae6a4c18c20799bf280aa1b0e74526110d3480035251a0923d5a1ac60 not found: ID does not exist" Jun 08 11:52:19.365068 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.365045 2569 scope.go:117] "RemoveContainer" containerID="fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b" Jun 08 11:52:19.365141 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.365099 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb"] Jun 08 11:52:19.368344 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.368325 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-s76nb"] Jun 08 11:52:19.372750 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.372736 2569 scope.go:117] "RemoveContainer" containerID="d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207" Jun 08 11:52:19.379996 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.379979 2569 scope.go:117] "RemoveContainer" containerID="804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9" Jun 08 11:52:19.387368 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.387201 2569 scope.go:117] "RemoveContainer" containerID="edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78" Jun 08 11:52:19.394238 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.394222 2569 scope.go:117] "RemoveContainer" containerID="d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b" Jun 08 11:52:19.401194 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.401168 2569 scope.go:117] "RemoveContainer" containerID="3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d" Jun 08 11:52:19.414209 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.414191 2569 scope.go:117] "RemoveContainer" containerID="fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b" Jun 08 11:52:19.414462 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:52:19.414442 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b\": container with ID starting with fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b not found: ID does not exist" containerID="fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b" Jun 08 11:52:19.414538 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.414469 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b"} err="failed to get container status \"fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b\": rpc error: code = NotFound desc = could not find container \"fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b\": container with ID starting with fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b not found: ID does not exist" Jun 08 11:52:19.414538 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.414489 2569 scope.go:117] "RemoveContainer" containerID="d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207" Jun 08 11:52:19.414775 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:52:19.414760 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207\": container with ID starting with d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207 not found: ID does not exist" containerID="d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207" Jun 08 11:52:19.414831 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.414781 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207"} err="failed to get container status \"d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207\": rpc error: code = NotFound desc = could not find container \"d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207\": container with ID starting with d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207 not found: ID does not exist" Jun 08 11:52:19.414831 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.414796 2569 scope.go:117] "RemoveContainer" containerID="804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9" Jun 08 11:52:19.415049 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:52:19.415033 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9\": container with ID starting with 804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9 not found: ID does not exist" containerID="804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9" Jun 08 11:52:19.415087 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.415057 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9"} err="failed to get container status \"804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9\": rpc error: code = NotFound desc = could not find container \"804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9\": container with ID starting with 804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9 not found: ID does not exist" Jun 08 11:52:19.415087 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.415072 2569 scope.go:117] "RemoveContainer" containerID="edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78" Jun 08 11:52:19.415271 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:52:19.415256 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78\": container with ID starting with edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78 not found: ID does not exist" containerID="edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78" Jun 08 11:52:19.415311 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.415276 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78"} err="failed to get container status \"edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78\": rpc error: code = NotFound desc = could not find container \"edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78\": container with ID starting with edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78 not found: ID does not exist" Jun 08 11:52:19.415311 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.415296 2569 scope.go:117] "RemoveContainer" containerID="d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b" Jun 08 11:52:19.415483 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:52:19.415468 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b\": container with ID starting with d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b not found: ID does not exist" containerID="d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b" Jun 08 11:52:19.415524 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.415485 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b"} err="failed to get container status \"d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b\": rpc error: code = NotFound desc = could not find container \"d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b\": container with ID starting with d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b not found: ID does not exist" Jun 08 11:52:19.415524 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.415497 2569 scope.go:117] "RemoveContainer" containerID="3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d" Jun 08 11:52:19.415717 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:52:19.415702 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d\": container with ID starting with 3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d not found: ID does not exist" containerID="3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d" Jun 08 11:52:19.415769 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.415721 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d"} err="failed to get container status \"3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d\": rpc error: code = NotFound desc = could not find container \"3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d\": container with ID starting with 3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d not found: ID does not exist" Jun 08 11:52:19.415769 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.415733 2569 scope.go:117] "RemoveContainer" containerID="fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b" Jun 08 11:52:19.415986 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.415966 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b"} err="failed to get container status \"fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b\": rpc error: code = NotFound desc = could not find container \"fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b\": container with ID starting with fb6a02a0b586ae0e8a0bce04e68d36609703a784dcf18246af59c3687f749c3b not found: ID does not exist" Jun 08 11:52:19.416048 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.415987 2569 scope.go:117] "RemoveContainer" containerID="d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207" Jun 08 11:52:19.416205 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.416188 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207"} err="failed to get container status \"d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207\": rpc error: code = NotFound desc = could not find container \"d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207\": container with ID starting with d65bcc640adbf5ad654435092a3a1ee9f65c03631e1c3e736c4d0e1a6ba7a207 not found: ID does not exist" Jun 08 11:52:19.416249 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.416207 2569 scope.go:117] "RemoveContainer" containerID="804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9" Jun 08 11:52:19.416452 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.416438 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9"} err="failed to get container status \"804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9\": rpc error: code = NotFound desc = could not find container \"804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9\": container with ID starting with 804a170ae5224c1089da4f25ca322c07ca54c53bc17b780737034bd65165b0d9 not found: ID does not exist" Jun 08 11:52:19.416494 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.416454 2569 scope.go:117] "RemoveContainer" containerID="edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78" Jun 08 11:52:19.416676 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.416658 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78"} err="failed to get container status \"edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78\": rpc error: code = NotFound desc = could not find container \"edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78\": container with ID starting with edb70d18b8b927ff289c96a8c6976a4bbcf690201c4b4142162f188fbb702f78 not found: ID does not exist" Jun 08 11:52:19.416723 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.416677 2569 scope.go:117] "RemoveContainer" containerID="d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b" Jun 08 11:52:19.416904 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.416889 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b"} err="failed to get container status \"d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b\": rpc error: code = NotFound desc = could not find container \"d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b\": container with ID starting with d35bc069f3c285be12fd7297d970cb5ac1e0ea6a7ad6c05eab5e6aa0d682540b not found: ID does not exist" Jun 08 11:52:19.416939 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.416905 2569 scope.go:117] "RemoveContainer" containerID="3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d" Jun 08 11:52:19.417113 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:19.417094 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d"} err="failed to get container status \"3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d\": rpc error: code = NotFound desc = could not find container \"3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d\": container with ID starting with 3a83cee182cd39dfc298d7630adfd5d73d7004066553f0d56b01328180cb960d not found: ID does not exist" Jun 08 11:52:20.953704 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:20.953671 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" path="/var/lib/kubelet/pods/4fdf7a6b-7a68-404c-8ec7-643212c1095e/volumes" Jun 08 11:52:20.954298 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:20.954283 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" path="/var/lib/kubelet/pods/e250596e-e1dc-4fb3-a66f-8e26a144d9a5/volumes" Jun 08 11:52:37.399500 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399464 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ccjvt/must-gather-kvlt9"] Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399791 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="feast-init" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399802 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="feast-init" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399814 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="feast-apply" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399820 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="feast-apply" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399828 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="offline" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399834 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="offline" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399844 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="online" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399849 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="online" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399856 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="ui" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399861 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="ui" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399867 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="feast-init" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399872 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="feast-init" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399879 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="offline" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399884 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="offline" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399892 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="feast-apply" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399898 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="feast-apply" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399905 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="registry" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399910 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="registry" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399920 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="ui" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399925 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="ui" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399933 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="online" Jun 08 11:52:37.399945 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399939 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="online" Jun 08 11:52:37.400760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399984 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="registry" Jun 08 11:52:37.400760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399992 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="online" Jun 08 11:52:37.400760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.399998 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="online" Jun 08 11:52:37.400760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.400003 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="ui" Jun 08 11:52:37.400760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.400009 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fdf7a6b-7a68-404c-8ec7-643212c1095e" containerName="offline" Jun 08 11:52:37.400760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.400015 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="ui" Jun 08 11:52:37.400760 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.400021 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e250596e-e1dc-4fb3-a66f-8e26a144d9a5" containerName="offline" Jun 08 11:52:37.403202 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.403187 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:52:37.405567 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.405541 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ccjvt\"/\"openshift-service-ca.crt\"" Jun 08 11:52:37.405710 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.405548 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ccjvt\"/\"kube-root-ca.crt\"" Jun 08 11:52:37.406433 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.406420 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ccjvt\"/\"default-dockercfg-hkqf9\"" Jun 08 11:52:37.412438 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.412419 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ccjvt/must-gather-kvlt9"] Jun 08 11:52:37.517373 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.517335 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c48e753-0a52-45f1-97e8-f93bcd608eb9-must-gather-output\") pod \"must-gather-kvlt9\" (UID: \"9c48e753-0a52-45f1-97e8-f93bcd608eb9\") " pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:52:37.517373 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.517374 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpqd7\" (UniqueName: \"kubernetes.io/projected/9c48e753-0a52-45f1-97e8-f93bcd608eb9-kube-api-access-zpqd7\") pod \"must-gather-kvlt9\" (UID: \"9c48e753-0a52-45f1-97e8-f93bcd608eb9\") " pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:52:37.618370 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.618333 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c48e753-0a52-45f1-97e8-f93bcd608eb9-must-gather-output\") pod \"must-gather-kvlt9\" (UID: \"9c48e753-0a52-45f1-97e8-f93bcd608eb9\") " pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:52:37.618370 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.618372 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpqd7\" (UniqueName: \"kubernetes.io/projected/9c48e753-0a52-45f1-97e8-f93bcd608eb9-kube-api-access-zpqd7\") pod \"must-gather-kvlt9\" (UID: \"9c48e753-0a52-45f1-97e8-f93bcd608eb9\") " pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:52:37.618708 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.618686 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c48e753-0a52-45f1-97e8-f93bcd608eb9-must-gather-output\") pod \"must-gather-kvlt9\" (UID: \"9c48e753-0a52-45f1-97e8-f93bcd608eb9\") " pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:52:37.627979 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.627956 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpqd7\" (UniqueName: \"kubernetes.io/projected/9c48e753-0a52-45f1-97e8-f93bcd608eb9-kube-api-access-zpqd7\") pod \"must-gather-kvlt9\" (UID: \"9c48e753-0a52-45f1-97e8-f93bcd608eb9\") " pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:52:37.721063 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.721036 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:52:37.853132 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.852941 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ccjvt/must-gather-kvlt9"] Jun 08 11:52:37.856278 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:52:37.856249 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c48e753_0a52_45f1_97e8_f93bcd608eb9.slice/crio-a50c72c8ebce6c9fe3e1624ff4e683b588c17f9e35e725525d50ac2054899b0a WatchSource:0}: Error finding container a50c72c8ebce6c9fe3e1624ff4e683b588c17f9e35e725525d50ac2054899b0a: Status 404 returned error can't find the container with id a50c72c8ebce6c9fe3e1624ff4e683b588c17f9e35e725525d50ac2054899b0a Jun 08 11:52:37.858496 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:37.858478 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 08 11:52:38.405176 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:38.405146 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" event={"ID":"9c48e753-0a52-45f1-97e8-f93bcd608eb9","Type":"ContainerStarted","Data":"a50c72c8ebce6c9fe3e1624ff4e683b588c17f9e35e725525d50ac2054899b0a"} Jun 08 11:52:44.448070 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:44.447960 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" event={"ID":"9c48e753-0a52-45f1-97e8-f93bcd608eb9","Type":"ContainerStarted","Data":"dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504"} Jun 08 11:52:44.448070 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:44.448016 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" event={"ID":"9c48e753-0a52-45f1-97e8-f93bcd608eb9","Type":"ContainerStarted","Data":"205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a"} Jun 08 11:52:44.466627 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:44.466563 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" podStartSLOduration=1.296452247 podStartE2EDuration="7.466546294s" totalImagesPullingTime="6.170094047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:52:44.464492387 +0000 UTC m=+1953.948891250" watchObservedRunningTime="2026-06-08 11:52:44.466546294 +0000 UTC m=+1953.950945222" Jun 08 11:52:54.498478 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:54.498443 2569 generic.go:356] "Generic (PLEG): container finished" podID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" containerID="205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a" exitCode=0 Jun 08 11:52:54.498928 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:54.498515 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" event={"ID":"9c48e753-0a52-45f1-97e8-f93bcd608eb9","Type":"ContainerDied","Data":"205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a"} Jun 08 11:52:54.498928 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:54.498824 2569 scope.go:117] "RemoveContainer" containerID="205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a" Jun 08 11:52:54.667566 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:54.667529 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ccjvt_must-gather-kvlt9_9c48e753-0a52-45f1-97e8-f93bcd608eb9/gather/0.log" Jun 08 11:52:57.684052 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:57.684014 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dvnth_579ebcf9-8227-48b0-ac5a-1d9e2b8504fb/global-pull-secret-syncer/0.log" Jun 08 11:52:57.853700 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:57.853669 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hjb7c_b0219d35-4a8d-4383-acda-cc5fad912ade/konnectivity-agent/0.log" Jun 08 11:52:57.906304 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:52:57.906266 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-94.ec2.internal_b69eb2fa1fe4d666208120158aec889b/haproxy/0.log" Jun 08 11:53:00.009758 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.009724 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-ccjvt/must-gather-kvlt9"] Jun 08 11:53:00.010226 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.009909 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" containerName="copy" containerID="cri-o://dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504" gracePeriod=2 Jun 08 11:53:00.014284 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.014226 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-ccjvt/must-gather-kvlt9"] Jun 08 11:53:00.250458 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.250429 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ccjvt_must-gather-kvlt9_9c48e753-0a52-45f1-97e8-f93bcd608eb9/copy/0.log" Jun 08 11:53:00.252079 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.252061 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:53:00.254059 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:53:00.254032 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kvlt9\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ccjvt\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:53:00.313373 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.313277 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c48e753-0a52-45f1-97e8-f93bcd608eb9-must-gather-output\") pod \"9c48e753-0a52-45f1-97e8-f93bcd608eb9\" (UID: \"9c48e753-0a52-45f1-97e8-f93bcd608eb9\") " Jun 08 11:53:00.313534 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.313409 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpqd7\" (UniqueName: \"kubernetes.io/projected/9c48e753-0a52-45f1-97e8-f93bcd608eb9-kube-api-access-zpqd7\") pod \"9c48e753-0a52-45f1-97e8-f93bcd608eb9\" (UID: \"9c48e753-0a52-45f1-97e8-f93bcd608eb9\") " Jun 08 11:53:00.313667 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.313644 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c48e753-0a52-45f1-97e8-f93bcd608eb9-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "9c48e753-0a52-45f1-97e8-f93bcd608eb9" (UID: "9c48e753-0a52-45f1-97e8-f93bcd608eb9"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:53:00.315700 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.315671 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c48e753-0a52-45f1-97e8-f93bcd608eb9-kube-api-access-zpqd7" (OuterVolumeSpecName: "kube-api-access-zpqd7") pod "9c48e753-0a52-45f1-97e8-f93bcd608eb9" (UID: "9c48e753-0a52-45f1-97e8-f93bcd608eb9"). InnerVolumeSpecName "kube-api-access-zpqd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:53:00.414404 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.414366 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpqd7\" (UniqueName: \"kubernetes.io/projected/9c48e753-0a52-45f1-97e8-f93bcd608eb9-kube-api-access-zpqd7\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:53:00.414404 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.414397 2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c48e753-0a52-45f1-97e8-f93bcd608eb9-must-gather-output\") on node \"ip-10-0-130-94.ec2.internal\" DevicePath \"\"" Jun 08 11:53:00.526197 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.526169 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ccjvt_must-gather-kvlt9_9c48e753-0a52-45f1-97e8-f93bcd608eb9/copy/0.log" Jun 08 11:53:00.526522 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.526496 2569 generic.go:356] "Generic (PLEG): container finished" podID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" containerID="dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504" exitCode=143 Jun 08 11:53:00.526665 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.526544 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:53:00.526665 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.526605 2569 scope.go:117] "RemoveContainer" containerID="dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504" Jun 08 11:53:00.528838 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:53:00.528800 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kvlt9\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ccjvt\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:53:00.535074 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.535060 2569 scope.go:117] "RemoveContainer" containerID="205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a" Jun 08 11:53:00.536468 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:53:00.536444 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kvlt9\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ccjvt\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:53:00.545743 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.545722 2569 scope.go:117] "RemoveContainer" containerID="dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504" Jun 08 11:53:00.546013 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:53:00.545994 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504\": container with ID starting with dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504 not found: ID does not exist" containerID="dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504" Jun 08 11:53:00.546086 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.546021 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504"} err="failed to get container status \"dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504\": rpc error: code = NotFound desc = could not find container \"dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504\": container with ID starting with dbed39072d335d310d6b18e8f31f75caff2d9ef22b9d9250ee0616f3a889e504 not found: ID does not exist" Jun 08 11:53:00.546086 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.546038 2569 scope.go:117] "RemoveContainer" containerID="205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a" Jun 08 11:53:00.546309 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:53:00.546292 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a\": container with ID starting with 205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a not found: ID does not exist" containerID="205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a" Jun 08 11:53:00.546356 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.546316 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a"} err="failed to get container status \"205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a\": rpc error: code = NotFound desc = could not find container \"205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a\": container with ID starting with 205102ceaa4d5537bda4a2ff928df6fafeb624870bdc145fb3d621acae1a8c1a not found: ID does not exist" Jun 08 11:53:00.953522 ip-10-0-130-94 kubenswrapper[2569]: E0608 11:53:00.953487 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kvlt9\" is forbidden: User \"system:node:ip-10-0-130-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ccjvt\": no relationship found between node 'ip-10-0-130-94.ec2.internal' and this object" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" pod="openshift-must-gather-ccjvt/must-gather-kvlt9" Jun 08 11:53:00.953704 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:00.953551 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" path="/var/lib/kubelet/pods/9c48e753-0a52-45f1-97e8-f93bcd608eb9/volumes" Jun 08 11:53:01.550237 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:01.550211 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7302a48-bc54-494e-bdea-b6f958088c7b/alertmanager/0.log" Jun 08 11:53:01.577223 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:01.577199 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7302a48-bc54-494e-bdea-b6f958088c7b/config-reloader/0.log" Jun 08 11:53:01.601969 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:01.601952 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7302a48-bc54-494e-bdea-b6f958088c7b/kube-rbac-proxy-web/0.log" Jun 08 11:53:01.628579 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:01.628556 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7302a48-bc54-494e-bdea-b6f958088c7b/kube-rbac-proxy/0.log" Jun 08 11:53:01.662828 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:01.662813 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7302a48-bc54-494e-bdea-b6f958088c7b/kube-rbac-proxy-metric/0.log" Jun 08 11:53:01.692815 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:01.692798 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7302a48-bc54-494e-bdea-b6f958088c7b/prom-label-proxy/0.log" Jun 08 11:53:01.724148 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:01.724127 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7302a48-bc54-494e-bdea-b6f958088c7b/init-config-reloader/0.log" Jun 08 11:53:01.887368 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:01.887283 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7547496f79-twqhk_db77db9c-9324-4af1-b6fa-c01ed356156c/metrics-server/0.log" Jun 08 11:53:02.227436 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.227410 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-twz5m_0becfb22-31da-4d52-bd68-a80466b75ad0/node-exporter/0.log" Jun 08 11:53:02.250157 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.250133 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-twz5m_0becfb22-31da-4d52-bd68-a80466b75ad0/kube-rbac-proxy/0.log" Jun 08 11:53:02.273884 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.273860 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-twz5m_0becfb22-31da-4d52-bd68-a80466b75ad0/init-textfile/0.log" Jun 08 11:53:02.584819 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.584741 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7f7d445d84-gfg9p_e096e280-68d9-465a-8e52-562730844ab5/prometheus-operator/0.log" Jun 08 11:53:02.612797 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.612768 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7f7d445d84-gfg9p_e096e280-68d9-465a-8e52-562730844ab5/kube-rbac-proxy/0.log" Jun 08 11:53:02.638840 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.638800 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5b6b8f594-zsdh7_c74c35a7-f89f-45b9-b8c0-ca6f2f91c562/prometheus-operator-admission-webhook/0.log" Jun 08 11:53:02.672299 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.672270 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58c8648587-jzgdp_79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6/telemeter-client/0.log" Jun 08 11:53:02.699354 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.699335 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58c8648587-jzgdp_79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6/reload/0.log" Jun 08 11:53:02.725574 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.725553 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58c8648587-jzgdp_79f6dcb5-59cc-4b66-87c5-eb1cac2c86f6/kube-rbac-proxy/0.log" Jun 08 11:53:02.765072 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.765048 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/thanos-query/0.log" Jun 08 11:53:02.790628 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.790569 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/kube-rbac-proxy-web/0.log" Jun 08 11:53:02.831119 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.831082 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/kube-rbac-proxy/0.log" Jun 08 11:53:02.862542 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.862471 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/prom-label-proxy/0.log" Jun 08 11:53:02.891589 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.891557 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/kube-rbac-proxy-rules/0.log" Jun 08 11:53:02.920152 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:02.920132 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-694649bbf5-gvrsm_52a4c1ce-25bb-4193-bf2b-4f1322fa0d55/kube-rbac-proxy-metrics/0.log" Jun 08 11:53:04.125946 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:04.125903 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d5d5bb78b-cnm4d_ef8c72d1-5ce7-44c0-a259-cf8b40983674/networking-console-plugin/0.log" Jun 08 11:53:05.003285 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.003250 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6657875f97-g6lcj_66448ff5-45a3-4749-b993-e446af7d3dd6/download-server/0.log" Jun 08 11:53:05.395522 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.395450 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr"] Jun 08 11:53:05.395899 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.395792 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" containerName="gather" Jun 08 11:53:05.395899 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.395804 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" containerName="gather" Jun 08 11:53:05.395899 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.395817 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" containerName="copy" Jun 08 11:53:05.395899 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.395822 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" containerName="copy" Jun 08 11:53:05.395899 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.395876 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" containerName="copy" Jun 08 11:53:05.395899 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.395887 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c48e753-0a52-45f1-97e8-f93bcd608eb9" containerName="gather" Jun 08 11:53:05.398911 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.398895 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.401105 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.401081 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tgp8n\"/\"default-dockercfg-n6rcm\"" Jun 08 11:53:05.401214 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.401081 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tgp8n\"/\"kube-root-ca.crt\"" Jun 08 11:53:05.401997 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.401980 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tgp8n\"/\"openshift-service-ca.crt\"" Jun 08 11:53:05.409084 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.409065 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr"] Jun 08 11:53:05.450987 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.450957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-lib-modules\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.450987 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.450991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-podres\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.451168 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.451073 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-sys\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.451168 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.451119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8k6w\" (UniqueName: \"kubernetes.io/projected/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-kube-api-access-g8k6w\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.451234 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.451166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-proc\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.505005 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.504973 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78dd696465-lcrhz_8341efd3-6f21-47f9-9b2c-bad74ab7935d/volume-data-source-validator/0.log" Jun 08 11:53:05.551526 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.551497 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-sys\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.551709 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.551536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8k6w\" (UniqueName: \"kubernetes.io/projected/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-kube-api-access-g8k6w\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.551709 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.551646 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-sys\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.551709 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.551670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-proc\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.551709 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.551705 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-lib-modules\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.551899 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.551752 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-podres\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.551899 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.551783 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-proc\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.551899 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.551817 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-lib-modules\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.552009 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.551894 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-podres\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.559852 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.559826 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8k6w\" (UniqueName: \"kubernetes.io/projected/0fd4bca9-0c3e-4211-a83c-7ba0226c0d03-kube-api-access-g8k6w\") pod \"perf-node-gather-daemonset-w7nzr\" (UID: \"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03\") " pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.709295 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.709201 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:05.840936 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:05.840911 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr"] Jun 08 11:53:05.844027 ip-10-0-130-94 kubenswrapper[2569]: W0608 11:53:05.843997 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0fd4bca9_0c3e_4211_a83c_7ba0226c0d03.slice/crio-a558fe91dd16f32c0c9e2179753664b5291fc6f02e52d6e171a244defff5a227 WatchSource:0}: Error finding container a558fe91dd16f32c0c9e2179753664b5291fc6f02e52d6e171a244defff5a227: Status 404 returned error can't find the container with id a558fe91dd16f32c0c9e2179753664b5291fc6f02e52d6e171a244defff5a227 Jun 08 11:53:06.359195 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:06.359164 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n5m8b_483118b0-02a0-4b58-baf7-2e16a4fb6e2e/dns/0.log" Jun 08 11:53:06.379625 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:06.379594 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n5m8b_483118b0-02a0-4b58-baf7-2e16a4fb6e2e/kube-rbac-proxy/0.log" Jun 08 11:53:06.508581 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:06.508553 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bkxdh_822bda63-ed87-40ff-86a1-5ebee377ed5a/dns-node-resolver/0.log" Jun 08 11:53:06.551410 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:06.551378 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" event={"ID":"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03","Type":"ContainerStarted","Data":"a182720567277131927ff234c3b6f3dc6c31cee772185fab12662f57b104f6b2"} Jun 08 11:53:06.551589 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:06.551431 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" event={"ID":"0fd4bca9-0c3e-4211-a83c-7ba0226c0d03","Type":"ContainerStarted","Data":"a558fe91dd16f32c0c9e2179753664b5291fc6f02e52d6e171a244defff5a227"} Jun 08 11:53:06.551589 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:06.551485 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:06.570883 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:06.570835 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" podStartSLOduration=1.570822325 podStartE2EDuration="1.570822325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:53:06.56946405 +0000 UTC m=+1976.053862914" watchObservedRunningTime="2026-06-08 11:53:06.570822325 +0000 UTC m=+1976.055221188" Jun 08 11:53:06.920339 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:06.920305 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65949fcfbf-49m7t_177e0d64-9385-4ba2-a01f-d825fa7fddbf/registry/0.log" Jun 08 11:53:06.926504 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:06.926482 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65949fcfbf-49m7t_177e0d64-9385-4ba2-a01f-d825fa7fddbf/registry/1.log" Jun 08 11:53:06.993192 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:06.993168 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zmxqf_25861972-bb71-4dc8-bfae-31d8c5a5bd8e/node-ca/0.log" Jun 08 11:53:07.636342 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:07.636315 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f975c8c69-c85sm_2bcc00d8-28dc-4f53-ba05-4b53c852e037/router/0.log" Jun 08 11:53:08.000796 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:08.000764 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jjjwt_74ad6f9e-7b62-45d8-80b6-391b0f3758cb/serve-healthcheck-canary/0.log" Jun 08 11:53:08.319726 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:08.319630 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-896549976-8l62d_fc8847f9-8e90-498b-9c06-237949ccad50/insights-operator/0.log" Jun 08 11:53:08.321640 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:08.321604 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-896549976-8l62d_fc8847f9-8e90-498b-9c06-237949ccad50/insights-operator/1.log" Jun 08 11:53:08.412783 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:08.412754 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qnqgr_df10cd2f-a986-4b4d-bca6-f2fb37457b18/kube-rbac-proxy/0.log" Jun 08 11:53:08.436049 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:08.436027 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qnqgr_df10cd2f-a986-4b4d-bca6-f2fb37457b18/exporter/0.log" Jun 08 11:53:08.460994 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:08.460968 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qnqgr_df10cd2f-a986-4b4d-bca6-f2fb37457b18/extractor/0.log" Jun 08 11:53:12.565604 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:12.565576 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tgp8n/perf-node-gather-daemonset-w7nzr" Jun 08 11:53:13.658393 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:13.658364 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6qdwq_80d9b28a-6045-49bf-8f48-304a8396799f/kube-multus-additional-cni-plugins/0.log" Jun 08 11:53:13.680382 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:13.680354 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6qdwq_80d9b28a-6045-49bf-8f48-304a8396799f/egress-router-binary-copy/0.log" Jun 08 11:53:13.700779 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:13.700757 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6qdwq_80d9b28a-6045-49bf-8f48-304a8396799f/cni-plugins/0.log" Jun 08 11:53:13.722254 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:13.722228 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6qdwq_80d9b28a-6045-49bf-8f48-304a8396799f/bond-cni-plugin/0.log" Jun 08 11:53:13.746403 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:13.746375 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6qdwq_80d9b28a-6045-49bf-8f48-304a8396799f/routeoverride-cni/0.log" Jun 08 11:53:13.772187 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:13.772160 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6qdwq_80d9b28a-6045-49bf-8f48-304a8396799f/whereabouts-cni-bincopy/0.log" Jun 08 11:53:13.795472 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:13.795450 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6qdwq_80d9b28a-6045-49bf-8f48-304a8396799f/whereabouts-cni/0.log" Jun 08 11:53:14.173088 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:14.173064 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hg6x5_be0f4959-65ba-40a2-aca1-76b571c2ce30/kube-multus/0.log" Jun 08 11:53:14.355846 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:14.355763 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kthbk_2413fc4e-6ee1-47a8-8621-657e43ccfcf4/network-metrics-daemon/0.log" Jun 08 11:53:14.377096 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:14.377069 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kthbk_2413fc4e-6ee1-47a8-8621-657e43ccfcf4/kube-rbac-proxy/0.log" Jun 08 11:53:15.397084 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:15.396998 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx9jt_6534a1c3-5859-4fc5-8b05-ca877b16aa77/ovn-controller/0.log" Jun 08 11:53:15.414649 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:15.414601 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx9jt_6534a1c3-5859-4fc5-8b05-ca877b16aa77/ovn-acl-logging/0.log" Jun 08 11:53:15.439038 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:15.439006 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx9jt_6534a1c3-5859-4fc5-8b05-ca877b16aa77/kube-rbac-proxy-node/0.log" Jun 08 11:53:15.458868 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:15.458842 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx9jt_6534a1c3-5859-4fc5-8b05-ca877b16aa77/kube-rbac-proxy-ovn-metrics/0.log" Jun 08 11:53:15.477140 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:15.477114 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx9jt_6534a1c3-5859-4fc5-8b05-ca877b16aa77/northd/0.log" Jun 08 11:53:15.497364 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:15.497338 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx9jt_6534a1c3-5859-4fc5-8b05-ca877b16aa77/nbdb/0.log" Jun 08 11:53:15.518077 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:15.518054 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx9jt_6534a1c3-5859-4fc5-8b05-ca877b16aa77/sbdb/0.log" Jun 08 11:53:15.559845 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:15.559821 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx9jt_6534a1c3-5859-4fc5-8b05-ca877b16aa77/ovnkube-controller/0.log" Jun 08 11:53:16.766147 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:16.766116 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-dbd9cf844-47jkz_5bd062bd-d5b9-411a-8a07-52e26d6e2eca/check-endpoints/0.log" Jun 08 11:53:16.827661 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:16.827568 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-m6fn6_f8064e28-3a81-423c-b1b4-3610c44f0f46/network-check-target-container/0.log" Jun 08 11:53:17.680815 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:17.680786 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-k2t8t_74c2edb1-405d-4d3d-ae3f-d07b87d495f9/iptables-alerter/0.log" Jun 08 11:53:18.280263 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:18.280232 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-2m76c_722e256c-433a-4b5f-aba9-960dd03efc28/tuned/0.log" Jun 08 11:53:19.968033 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:19.967989 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-854f8b96b6-dzqjw_fce44b84-bbb6-4d3a-9679-02bfa2d73a35/cluster-samples-operator/0.log" Jun 08 11:53:19.984973 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:19.984944 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-854f8b96b6-dzqjw_fce44b84-bbb6-4d3a-9679-02bfa2d73a35/cluster-samples-operator-watch/0.log" Jun 08 11:53:21.594051 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:21.594020 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-htqfd_08095808-76be-4f3c-8ff2-c7dfaeb5c832/csi-driver/0.log" Jun 08 11:53:21.618935 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:21.618906 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-htqfd_08095808-76be-4f3c-8ff2-c7dfaeb5c832/csi-node-driver-registrar/0.log" Jun 08 11:53:21.644082 ip-10-0-130-94 kubenswrapper[2569]: I0608 11:53:21.644055 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-htqfd_08095808-76be-4f3c-8ff2-c7dfaeb5c832/csi-liveness-probe/0.log"