Sep 09 17:18:44.255474 ip-10-0-141-213 systemd[1]: Starting Kubernetes Kubelet... Sep 09 17:18:44.636432 ip-10-0-141-213 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 17:18:44.636432 ip-10-0-141-213 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 17:18:44.636432 ip-10-0-141-213 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 17:18:44.636432 ip-10-0-141-213 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 17:18:44.636432 ip-10-0-141-213 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 17:18:44.637097 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.636487 2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 17:18:44.640955 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640939 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:18:44.640955 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640954 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640957 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640961 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640964 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640967 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640970 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640975 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640987 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640991 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640993 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640996 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.640999 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641002 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641004 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641007 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641010 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641012 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641015 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641017 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641020 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:18:44.641019 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641023 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641025 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641028 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641032 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641034 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641037 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641040 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641042 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641045 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641047 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641050 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641052 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641054 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641057 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641059 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641062 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641064 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641067 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641069 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641072 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:18:44.641557 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641075 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641078 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641080 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641083 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641085 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641088 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641090 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641093 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641095 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641098 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641100 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641103 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641107 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641109 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641113 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641115 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641118 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641120 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641123 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641126 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:18:44.642305 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641130 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641133 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641136 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641139 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641142 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641144 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641147 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641150 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641152 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641155 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641157 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641160 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641162 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641165 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641168 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641170 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641173 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641175 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641178 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:18:44.642831 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641180 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641184 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641187 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641190 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641193 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641214 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641219 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641224 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641227 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641229 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641232 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641234 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641237 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641240 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641242 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641245 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641248 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641251 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:18:44.643325 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.641253 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:18:44.644193 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644176 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:18:44.644239 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644193 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:18:44.644239 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644213 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:18:44.644239 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644216 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:18:44.644239 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644219 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:18:44.644239 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644222 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:18:44.644239 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644225 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:18:44.644239 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644228 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:18:44.644239 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644231 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:18:44.644239 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644235 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:18:44.644239 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644238 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:18:44.644239 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644243 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644247 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644250 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644253 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644256 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644259 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644262 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644265 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644267 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644270 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644273 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644275 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644278 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644280 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644283 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644285 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644288 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644291 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644293 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644296 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:18:44.644507 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644298 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644301 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644304 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644307 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644309 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644312 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644316 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644319 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644322 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644325 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644327 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644330 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644332 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644336 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644340 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644342 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644345 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644348 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644351 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:18:44.645012 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644354 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644359 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644363 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644366 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644370 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644372 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644375 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644379 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644381 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644384 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644387 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644391 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644394 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644396 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644399 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644402 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644404 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644407 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644410 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644412 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:18:44.645502 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644415 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644417 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644420 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644422 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644425 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644427 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644430 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644434 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644437 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644439 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644442 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644444 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644447 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644450 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644453 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644455 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644458 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644460 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644463 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644465 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:18:44.646024 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644468 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644471 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644473 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644476 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644479 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644481 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644483 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644486 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644489 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644884 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644889 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644892 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644895 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644898 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644901 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644904 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644906 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644909 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644911 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644915 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:18:44.646539 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644917 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644920 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644923 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644926 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644928 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644931 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644934 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644937 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644940 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644944 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644947 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644950 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644953 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644955 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644958 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644960 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644963 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644966 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644968 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644971 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:18:44.647026 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644973 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644976 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644978 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644981 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644983 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644986 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644988 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644991 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644993 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.644996 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645000 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645004 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645008 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645011 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645014 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645017 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645020 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645022 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645025 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645028 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:18:44.647590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645031 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645033 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645035 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645038 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645040 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645043 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645046 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645048 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645052 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645055 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645059 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645062 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645065 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645068 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645071 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645073 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645076 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645078 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645080 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:18:44.648080 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645083 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645085 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645088 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645092 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645096 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645099 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645101 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645104 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645107 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645109 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645111 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645114 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645117 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645120 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645123 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645125 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645128 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645130 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645133 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645135 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:18:44.648565 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645138 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645140 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645143 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645146 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645148 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645150 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645153 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645155 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645158 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645240 2553 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645278 2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645285 2553 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645289 2553 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645293 2553 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645296 2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645302 2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645306 2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645309 2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645312 2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645316 2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645319 2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645322 2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 17:18:44.649058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645325 2553 flags.go:64] FLAG: --cgroup-root="" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645328 2553 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645330 2553 flags.go:64] FLAG: --client-ca-file="" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645334 2553 flags.go:64] FLAG: --cloud-provider="external" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645337 2553 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645342 2553 flags.go:64] FLAG: --cluster-domain="" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645345 2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645347 2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645351 2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645354 2553 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645358 2553 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645360 2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645363 2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645366 2553 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645369 2553 flags.go:64] FLAG: --contention-profiling="false" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645372 2553 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645375 2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645378 2553 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645381 2553 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645385 2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645388 2553 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645391 2553 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645394 2553 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645397 2553 flags.go:64] FLAG: --enable-server="true" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645400 2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 17:18:44.649598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645405 2553 flags.go:64] FLAG: --event-burst="100" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645408 2553 flags.go:64] FLAG: --event-qps="50" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645411 2553 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645414 2553 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645417 2553 flags.go:64] FLAG: --eviction-hard="" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645421 2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645424 2553 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645427 2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645429 2553 flags.go:64] FLAG: --eviction-soft="" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645433 2553 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645435 2553 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645439 2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645442 2553 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645445 2553 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645448 2553 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645451 2553 flags.go:64] FLAG: --feature-gates="" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645455 2553 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645458 2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645461 2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645463 2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645466 2553 flags.go:64] FLAG: --healthz-port="10248" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645469 2553 flags.go:64] FLAG: --help="false" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645472 2553 flags.go:64] FLAG: --hostname-override="ip-10-0-141-213.ec2.internal" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645475 2553 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 17:18:44.650216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645478 2553 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645481 2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645484 2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645488 2553 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645491 2553 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645494 2553 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645496 2553 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645499 2553 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645503 2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645507 2553 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645510 2553 flags.go:64] FLAG: --kube-reserved="" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645513 2553 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645516 2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645519 2553 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645521 2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645525 2553 flags.go:64] FLAG: --lock-file="" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645528 2553 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645531 2553 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645534 2553 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645539 2553 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645542 2553 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645545 2553 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645548 2553 flags.go:64] FLAG: --logging-format="text" Sep 09 17:18:44.650857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645550 2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645554 2553 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645556 2553 flags.go:64] FLAG: --manifest-url="" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645559 2553 flags.go:64] FLAG: --manifest-url-header="" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645563 2553 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645566 2553 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645570 2553 flags.go:64] FLAG: --max-pods="110" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645573 2553 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645575 2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645578 2553 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645581 2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645584 2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645587 2553 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645590 2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645597 2553 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645600 2553 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645603 2553 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645607 2553 flags.go:64] FLAG: --pod-cidr="" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645610 2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645615 2553 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645618 2553 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645621 2553 flags.go:64] FLAG: --pods-per-core="0" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645624 2553 flags.go:64] FLAG: --port="10250" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645627 2553 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 17:18:44.651440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645630 2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00d3459331902c04f" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645633 2553 flags.go:64] FLAG: --qos-reserved="" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645636 2553 flags.go:64] FLAG: --read-only-port="10255" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645639 2553 flags.go:64] FLAG: --register-node="true" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645641 2553 flags.go:64] FLAG: --register-with-taints="" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645645 2553 flags.go:64] FLAG: --registry-burst="10" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645649 2553 flags.go:64] FLAG: --registry-qps="5" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645652 2553 flags.go:64] FLAG: --reserved-cpus="" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645654 2553 flags.go:64] FLAG: --reserved-memory="" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645660 2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645663 2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645666 2553 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645669 2553 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645672 2553 flags.go:64] FLAG: --runonce="false" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645674 2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645677 2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645680 2553 flags.go:64] FLAG: --seccomp-default="false" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645683 2553 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645686 2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645689 2553 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645692 2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645695 2553 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645698 2553 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645701 2553 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645703 2553 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645707 2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 17:18:44.652062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645710 2553 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645713 2553 flags.go:64] FLAG: --system-cgroups="" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645716 2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645721 2553 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645724 2553 flags.go:64] FLAG: --tls-cert-file="" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645727 2553 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645731 2553 flags.go:64] FLAG: --tls-min-version="" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645734 2553 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645736 2553 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645739 2553 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645742 2553 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645745 2553 flags.go:64] FLAG: --v="2" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645748 2553 flags.go:64] FLAG: --version="false" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645752 2553 flags.go:64] FLAG: --vmodule="" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645756 2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.645759 2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645871 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645874 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645877 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645881 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645883 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645886 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645889 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:18:44.652756 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645893 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645896 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645898 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645901 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645904 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645906 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645909 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645911 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645916 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645919 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645921 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645924 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645926 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645929 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645932 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645934 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645937 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645939 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645943 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:18:44.653324 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645947 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645950 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645953 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645956 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645959 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645962 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645964 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645966 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645969 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645971 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645974 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645977 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645980 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645984 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645986 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645989 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645991 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645994 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645997 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.645999 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:18:44.653790 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646002 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646006 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646009 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646011 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646014 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646016 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646020 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646024 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646027 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646030 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646032 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646035 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646038 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646041 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646044 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646046 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646049 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646052 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646055 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:18:44.654303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646057 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646060 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646062 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646065 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646068 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646072 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646077 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646080 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646082 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646085 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646088 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646090 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646093 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646096 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646099 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646102 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646104 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646107 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646118 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646122 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:18:44.654754 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646125 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646128 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646130 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646133 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646136 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646138 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646141 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646143 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646146 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646148 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646151 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646155 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646158 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.646160 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.646778 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 17:18:44.655352 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.654011 2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.654027 2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.654040 2553 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.654046 2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654101 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654105 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654108 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654120 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654124 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654126 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654129 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654132 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654135 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654138 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654140 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654143 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654146 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654149 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654153 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:18:44.655738 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654157 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654160 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654163 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654166 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654168 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654171 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654174 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654177 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654180 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654184 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654187 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654190 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654193 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654218 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654223 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654226 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654228 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654231 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654233 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654236 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:18:44.656191 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654239 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654241 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654244 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654247 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654249 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654252 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654255 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654257 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654261 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654265 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654268 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654270 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654273 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654276 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654278 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654281 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654284 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654286 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:18:44.656690 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654289 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654291 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654294 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654297 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654300 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654303 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654305 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654308 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654311 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654314 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654317 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654320 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654322 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654326 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654329 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654332 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654334 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654336 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654339 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654342 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:18:44.657115 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654344 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654347 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654349 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654352 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654354 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654357 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654359 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654362 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654364 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654367 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654369 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654372 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654374 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654377 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654379 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654382 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654384 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654394 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654397 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654400 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:18:44.657627 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654403 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654405 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654408 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654411 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654413 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654416 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.654421 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654530 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654534 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654537 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654539 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654542 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654544 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654547 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654550 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:18:44.658124 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654553 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654557 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654560 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654563 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654566 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654569 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654572 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654575 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654577 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654580 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654582 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654585 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654587 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654590 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654593 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654596 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654599 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654602 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654605 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654607 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:18:44.658545 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654610 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654613 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654615 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654618 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654620 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654623 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654625 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654627 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654630 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654632 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654635 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654638 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654640 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654642 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654645 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654647 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654650 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654652 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654655 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654657 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:18:44.659028 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654660 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654662 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654665 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654667 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654669 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654672 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654674 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654677 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654679 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654682 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654685 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654688 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654690 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654693 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654695 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654698 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654700 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654703 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654705 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654708 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:18:44.659532 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654710 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654713 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654715 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654718 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654720 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654722 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654725 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654727 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654730 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654732 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654735 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654737 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654740 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654742 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654745 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654747 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654750 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654752 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654755 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654758 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:18:44.660017 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654761 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654764 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654768 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654772 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654774 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654778 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654781 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654784 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654786 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654789 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:44.654791 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.654796 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.654891 2553 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 17:18:44.660514 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.657005 2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 17:18:44.660831 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.657886 2553 server.go:1019] "Starting client certificate rotation" Sep 09 17:18:44.660831 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.657981 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 17:18:44.660831 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.658017 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 17:18:44.660831 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.660128 2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 17:18:44.665722 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.665695 2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 17:18:44.676099 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.676077 2553 log.go:25] "Validated CRI v1 runtime API" Sep 09 17:18:44.678995 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.678982 2553 log.go:25] "Validated CRI v1 image API" Sep 09 17:18:44.679807 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.679794 2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 17:18:44.683274 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.683253 2553 fs.go:135] Filesystem UUIDs: map[47ca2162-3a4b-4922-9407-80a9dfa1b7b2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e2515ba6-6a6d-494c-80d4-57bceffdec78:/dev/nvme0n1p3] Sep 09 17:18:44.683356 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.683270 2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 17:18:44.685571 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.685553 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 17:18:44.688391 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.688284 2553 manager.go:216] Machine: {Timestamp:2026-09-09 17:18:44.687038522 +0000 UTC m=+0.324368481 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098867 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a76988b4456b73b189e9f29e0c296 SystemUUID:ec2a7698-8b44-56b7-3b18-9e9f29e0c296 BootID:e18a59c9-5993-428e-9a2c-d3be01c4259a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e2:ef:85:80:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e2:ef:85:80:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:e4:7b:8d:03:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 17:18:44.688391 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.688384 2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 17:18:44.688527 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.688458 2553 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 17:18:44.691495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.691470 2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 17:18:44.691625 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.691498 2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-213.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 17:18:44.692319 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.692309 2553 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 17:18:44.692355 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.692322 2553 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 17:18:44.692355 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.692337 2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 17:18:44.693064 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.693054 2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 17:18:44.694608 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.694597 2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 17:18:44.697982 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.697971 2553 state_mem.go:36] "Initialized new in-memory state store" Sep 09 17:18:44.698106 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.698096 2553 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 17:18:44.698156 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.698131 2553 kubelet.go:486] "Attempting to sync node with API server" Sep 09 17:18:44.698156 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.698142 2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 17:18:44.698156 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.698153 2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 17:18:44.698277 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.698164 2553 kubelet.go:398] "Adding apiserver pod source" Sep 09 17:18:44.698277 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.698173 2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 17:18:44.698535 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.698523 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 17:18:44.698577 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.698544 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 17:18:44.700549 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.700529 2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 17:18:44.703292 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.703267 2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 17:18:44.703371 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.703307 2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 17:18:44.704070 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704059 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 17:18:44.704109 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704075 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 17:18:44.704109 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704082 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 17:18:44.704109 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704089 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 17:18:44.704109 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704098 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 17:18:44.704109 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704104 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 17:18:44.704268 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704116 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 17:18:44.704268 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704121 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 17:18:44.704268 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704127 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 17:18:44.704268 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704132 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 17:18:44.704268 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704145 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 17:18:44.704268 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.704150 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 17:18:44.705575 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.705562 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 17:18:44.705575 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.705572 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 17:18:44.707807 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.707593 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 17:18:44.707999 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.707811 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-213.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 17:18:44.709852 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.709837 2553 server.go:1268] "Started kubelet" Sep 09 17:18:44.709971 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.709934 2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 17:18:44.710038 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.709975 2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 17:18:44.710038 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.710035 2553 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 17:18:44.710715 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.710686 2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-213.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 17:18:44.711152 ip-10-0-141-213 systemd[1]: Started Kubernetes Kubelet. Sep 09 17:18:44.711759 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.711742 2553 server.go:310] "Adding debug handlers to kubelet server" Sep 09 17:18:44.711925 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.711910 2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 17:18:44.714145 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.714123 2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 17:18:44.716164 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.715357 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-213.ec2.internal.18d3b6ab809f5e52 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-213.ec2.internal,UID:ip-10-0-141-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-213.ec2.internal,},FirstTimestamp:2026-09-09 17:18:44.709809746 +0000 UTC m=+0.347139703,LastTimestamp:2026-09-09 17:18:44.709809746 +0000 UTC m=+0.347139703,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-213.ec2.internal,}" Sep 09 17:18:44.718002 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.717981 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 17:18:44.718573 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.718555 2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 17:18:44.718852 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.718833 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:44.719018 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.718951 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gmjhl" Sep 09 17:18:44.719124 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.719056 2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 17:18:44.719124 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.719058 2553 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 17:18:44.719124 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.719090 2553 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 17:18:44.719308 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.719185 2553 reconstruct.go:97] "Volume reconstruction finished" Sep 09 17:18:44.719308 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.719192 2553 reconciler.go:29] "Reconciler: start to sync state" Sep 09 17:18:44.719978 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.719841 2553 factory.go:55] Registering systemd factory Sep 09 17:18:44.720072 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.720063 2553 factory.go:223] Registration of the systemd container factory successfully Sep 09 17:18:44.720704 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.720688 2553 factory.go:153] Registering CRI-O factory Sep 09 17:18:44.720812 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.720803 2553 factory.go:223] Registration of the crio container factory successfully Sep 09 17:18:44.720905 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.720882 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 17:18:44.720980 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.720962 2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-213.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 17:18:44.721050 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.720976 2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 17:18:44.721139 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.721061 2553 factory.go:103] Registering Raw factory Sep 09 17:18:44.721139 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.721076 2553 manager.go:1196] Started watching for new ooms in manager Sep 09 17:18:44.721486 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.721472 2553 manager.go:349] Starting recovery of all containers Sep 09 17:18:44.726511 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.726483 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gmjhl" Sep 09 17:18:44.728838 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.728819 2553 manager.go:354] Recovery completed Sep 09 17:18:44.736048 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.736033 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:18:44.737910 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.737891 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:18:44.737975 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.737930 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:18:44.737975 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.737947 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:18:44.739676 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.739621 2553 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 17:18:44.739676 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.739635 2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 17:18:44.739676 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.739653 2553 state_mem.go:36] "Initialized new in-memory state store" Sep 09 17:18:44.739807 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.739706 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:18:44.740418 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.740401 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:18:44.740495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.740426 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:18:44.740495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.740440 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:18:44.819374 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.819342 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:44.883638 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.883622 2553 policy_none.go:49] "None policy: Start" Sep 09 17:18:44.883638 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.883640 2553 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 17:18:44.883712 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.883650 2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 17:18:44.884746 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.884735 2553 policy_none.go:47] "Start" Sep 09 17:18:44.919412 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.919400 2553 manager.go:337] "Starting Device Plugin manager" Sep 09 17:18:44.919511 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.919443 2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 17:18:44.919511 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.919458 2553 server.go:87] "Starting device plugin registration server" Sep 09 17:18:44.919511 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.919466 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:44.919834 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.919821 2553 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 17:18:44.919873 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.919837 2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 17:18:44.919982 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.919958 2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 17:18:44.920071 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.920050 2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 17:18:44.920071 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.920057 2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 17:18:44.920608 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.920591 2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 17:18:44.920692 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.920635 2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:44.927060 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.927043 2553 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-213.ec2.internal\" not found" node="ip-10-0-141-213.ec2.internal" Sep 09 17:18:44.942677 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.942648 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 17:18:44.943775 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.943754 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 17:18:44.943846 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.943802 2553 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 17:18:44.943846 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.943823 2553 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 17:18:44.943925 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:44.943860 2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 17:18:44.946076 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:44.946050 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 17:18:45.020441 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.020409 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:18:45.024841 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.024825 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:18:45.024906 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.024859 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:18:45.024906 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.024873 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:18:45.024906 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.024902 2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.032702 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.032686 2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.032743 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:45.032707 2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-213.ec2.internal\": node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:45.044209 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.044182 2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal"] Sep 09 17:18:45.044290 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.044252 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:18:45.045516 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.045503 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:18:45.045583 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.045526 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:18:45.045583 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.045535 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:18:45.046744 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.046732 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:18:45.046887 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.046872 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.046934 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.046903 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:18:45.047404 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.047389 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:18:45.047464 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.047418 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:18:45.047464 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.047427 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:18:45.047464 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.047389 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:18:45.047554 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.047478 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:18:45.047554 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.047496 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:18:45.048312 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:45.048294 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:45.048545 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.048525 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.048619 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.048557 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:18:45.049211 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.049188 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:18:45.049273 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.049220 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:18:45.049273 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.049229 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:18:45.073854 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:45.073835 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-213.ec2.internal\" not found" node="ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.078084 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:45.078070 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-213.ec2.internal\" not found" node="ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.120956 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.120934 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a288a822be8e393a56ffcbb08fdf962-config\") pod \"kube-apiserver-proxy-ip-10-0-141-213.ec2.internal\" (UID: \"0a288a822be8e393a56ffcbb08fdf962\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.121045 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.120959 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39af5840e15752b8ef00c7c30eaa9a5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"39af5840e15752b8ef00c7c30eaa9a5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.121045 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.120976 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39af5840e15752b8ef00c7c30eaa9a5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"39af5840e15752b8ef00c7c30eaa9a5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.148747 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:45.148699 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:45.221180 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.221157 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39af5840e15752b8ef00c7c30eaa9a5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"39af5840e15752b8ef00c7c30eaa9a5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.221275 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.221184 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a288a822be8e393a56ffcbb08fdf962-config\") pod \"kube-apiserver-proxy-ip-10-0-141-213.ec2.internal\" (UID: \"0a288a822be8e393a56ffcbb08fdf962\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.221275 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.221219 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39af5840e15752b8ef00c7c30eaa9a5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"39af5840e15752b8ef00c7c30eaa9a5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.221275 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.221239 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39af5840e15752b8ef00c7c30eaa9a5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"39af5840e15752b8ef00c7c30eaa9a5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.221275 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.221243 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a288a822be8e393a56ffcbb08fdf962-config\") pod \"kube-apiserver-proxy-ip-10-0-141-213.ec2.internal\" (UID: \"0a288a822be8e393a56ffcbb08fdf962\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.221275 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.221243 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39af5840e15752b8ef00c7c30eaa9a5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal\" (UID: \"39af5840e15752b8ef00c7c30eaa9a5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.249265 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:45.249246 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:45.349715 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:45.349686 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:45.376802 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.376776 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.381988 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:45.381970 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" Sep 09 17:18:45.450324 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:45.450266 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:45.550723 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:45.550702 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:47.048630 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:47.048606 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:47.055763 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.055745 2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 17:18:47.055894 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.055878 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 17:18:47.126577 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.126507 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 17:13:46 +0000 UTC" deadline="2028-05-25 01:53:24.13184295 +0000 UTC" Sep 09 17:18:47.126577 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.126537 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14960h34m37.005310559s" Sep 09 17:18:47.149119 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:47.149095 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:47.249991 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:47.249969 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:47.267764 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.267742 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 17:18:47.350749 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:47.350724 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:47.433414 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:47.433383 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a288a822be8e393a56ffcbb08fdf962.slice/crio-4b5ca4a3765572c92f8ff475591987fffbc639db3550371d99058d58ddf3ffa4 WatchSource:0}: Error finding container 4b5ca4a3765572c92f8ff475591987fffbc639db3550371d99058d58ddf3ffa4: Status 404 returned error can't find the container with id 4b5ca4a3765572c92f8ff475591987fffbc639db3550371d99058d58ddf3ffa4 Sep 09 17:18:47.433666 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:47.433642 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39af5840e15752b8ef00c7c30eaa9a5a.slice/crio-fc9e1117a71f19b2e2906c2aa285a7d89f43468dbe8e8ce8eea1e831e00d0b6b WatchSource:0}: Error finding container fc9e1117a71f19b2e2906c2aa285a7d89f43468dbe8e8ce8eea1e831e00d0b6b: Status 404 returned error can't find the container with id fc9e1117a71f19b2e2906c2aa285a7d89f43468dbe8e8ce8eea1e831e00d0b6b Sep 09 17:18:47.439865 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.439750 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 17:18:47.451660 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:47.451642 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-213.ec2.internal\" not found" Sep 09 17:18:47.519116 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.519093 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 17:18:47.561383 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.561362 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 17:18:47.617007 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.616985 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" Sep 09 17:18:47.627163 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.627147 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 17:18:47.628079 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.628068 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" Sep 09 17:18:47.636594 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:47.636580 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 17:18:48.096406 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.096364 2553 apiserver.go:52] "Watching apiserver" Sep 09 17:18:48.106805 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.106782 2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 17:18:48.107326 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.107302 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78","openshift-cluster-node-tuning-operator/tuned-rvdl4","openshift-dns/node-resolver-54x4s","openshift-image-registry/node-ca-tt8b9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal","openshift-network-operator/iptables-alerter-xhtvk","kube-system/global-pull-secret-syncer-zlhb4","openshift-multus/multus-additional-cni-plugins-wdsct","openshift-multus/multus-zg2bq","openshift-multus/network-metrics-daemon-tc2sl","openshift-network-diagnostics/network-check-target-f6l6s","openshift-ovn-kubernetes/ovnkube-node-xkdmr","kube-system/konnectivity-agent-q7ctz"] Sep 09 17:18:48.110124 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.109268 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xhtvk" Sep 09 17:18:48.113300 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.112711 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.114732 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.114709 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.117443 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.116121 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 17:18:48.117443 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.116336 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 17:18:48.117443 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.116506 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 17:18:48.117443 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.116597 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 17:18:48.117443 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.116667 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vqgmw\"" Sep 09 17:18:48.117443 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.116705 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-54x4s" Sep 09 17:18:48.117810 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.117764 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tt8b9" Sep 09 17:18:48.119180 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.119015 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:18:48.119787 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.119567 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 17:18:48.121497 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.120216 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 17:18:48.121497 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.120403 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 17:18:48.121497 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.120592 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 17:18:48.121911 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.121862 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.122830 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.122036 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:48.122830 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.122107 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 17:18:48.122830 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.122126 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:18:48.122830 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.122407 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 17:18:48.122830 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.122654 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 17:18:48.122830 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.122706 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-th7gc\"" Sep 09 17:18:48.122830 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.122720 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ndcs5\"" Sep 09 17:18:48.123236 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.123215 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mcz4z\"" Sep 09 17:18:48.123304 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.123254 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 17:18:48.123373 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.123354 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 17:18:48.123443 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.123411 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 17:18:48.123493 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.123479 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 17:18:48.124092 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.123570 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 17:18:48.124241 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.124219 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k66kq\"" Sep 09 17:18:48.124241 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.124376 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 17:18:48.124241 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.124410 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rg65z\"" Sep 09 17:18:48.124241 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.124604 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 17:18:48.124241 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.124643 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 17:18:48.125134 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.124909 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 17:18:48.125134 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.125111 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 17:18:48.125641 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.125319 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 17:18:48.125641 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.125572 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126028 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-cni-bin\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126077 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/690ed990-fef1-46c2-b214-3d2ba5fb78c4-env-overrides\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126101 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-modprobe-d\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126138 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-kubernetes\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126199 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1028b34-ff23-48f2-b853-0b0cdf0fe595-iptables-alerter-script\") pod \"iptables-alerter-xhtvk\" (UID: \"b1028b34-ff23-48f2-b853-0b0cdf0fe595\") " pod="openshift-network-operator/iptables-alerter-xhtvk" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126223 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1028b34-ff23-48f2-b853-0b0cdf0fe595-host-slash\") pod \"iptables-alerter-xhtvk\" (UID: \"b1028b34-ff23-48f2-b853-0b0cdf0fe595\") " pod="openshift-network-operator/iptables-alerter-xhtvk" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126253 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-cni-netd\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126277 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxvcq\" (UniqueName: \"kubernetes.io/projected/690ed990-fef1-46c2-b214-3d2ba5fb78c4-kube-api-access-wxvcq\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126305 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-sysconfig\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126347 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-sysctl-d\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126422 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-sys\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126447 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-node-log\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126483 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-log-socket\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126506 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/690ed990-fef1-46c2-b214-3d2ba5fb78c4-ovn-node-metrics-cert\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126547 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-systemd\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.126586 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh8zt\" (UniqueName: \"kubernetes.io/projected/2a5ff466-6997-438f-8364-7277a690051d-kube-api-access-rh8zt\") pod \"node-ca-tt8b9\" (UID: \"2a5ff466-6997-438f-8364-7277a690051d\") " pod="openshift-image-registry/node-ca-tt8b9" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.127070 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fjr29\"" Sep 09 17:18:48.127888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.127304 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 17:18:48.128850 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.127483 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.130250 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.130790 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-systemd-units\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.130835 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-var-lib-openvswitch\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.130872 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-run-openvswitch\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.130915 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-var-lib-kubelet\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.130947 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b88c2732-169f-47a7-b117-b556624997e7-hosts-file\") pod \"node-resolver-54x4s\" (UID: \"b88c2732-169f-47a7-b117-b556624997e7\") " pod="openshift-dns/node-resolver-54x4s" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.130971 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88c2732-169f-47a7-b117-b556624997e7-tmp-dir\") pod \"node-resolver-54x4s\" (UID: \"b88c2732-169f-47a7-b117-b556624997e7\") " pod="openshift-dns/node-resolver-54x4s" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131003 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbchp\" (UniqueName: \"kubernetes.io/projected/b88c2732-169f-47a7-b117-b556624997e7-kube-api-access-pbchp\") pod \"node-resolver-54x4s\" (UID: \"b88c2732-169f-47a7-b117-b556624997e7\") " pod="openshift-dns/node-resolver-54x4s" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131061 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-run-netns\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131093 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-run-systemd\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131123 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131154 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/690ed990-fef1-46c2-b214-3d2ba5fb78c4-ovnkube-config\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131180 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/690ed990-fef1-46c2-b214-3d2ba5fb78c4-ovnkube-script-lib\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131209 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-tmp\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131240 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp4fx\" (UniqueName: \"kubernetes.io/projected/b1028b34-ff23-48f2-b853-0b0cdf0fe595-kube-api-access-jp4fx\") pod \"iptables-alerter-xhtvk\" (UID: \"b1028b34-ff23-48f2-b853-0b0cdf0fe595\") " pod="openshift-network-operator/iptables-alerter-xhtvk" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131271 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-slash\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.134659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131296 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a5ff466-6997-438f-8364-7277a690051d-host\") pod \"node-ca-tt8b9\" (UID: \"2a5ff466-6997-438f-8364-7277a690051d\") " pod="openshift-image-registry/node-ca-tt8b9" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131331 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-kubelet\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131360 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-etc-openvswitch\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131391 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-run-ovn\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131422 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131454 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-host\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131479 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-tuned\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131509 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a5ff466-6997-438f-8364-7277a690051d-serviceca\") pod \"node-ca-tt8b9\" (UID: \"2a5ff466-6997-438f-8364-7277a690051d\") " pod="openshift-image-registry/node-ca-tt8b9" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131544 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-sysctl-conf\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131573 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-run\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131601 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-lib-modules\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.131641 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnk2q\" (UniqueName: \"kubernetes.io/projected/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-kube-api-access-gnk2q\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.132232 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.132248 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.132719 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.132838 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c8v66\"" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.133436 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 17:18:48.135649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.133544 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hmqzx\"" Sep 09 17:18:48.137614 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.135676 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:48.137614 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.135830 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:18:48.137614 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.136272 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 17:18:48.138653 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.138277 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:48.138653 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.138358 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:18:48.150899 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.150796 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nzhfr" Sep 09 17:18:48.157910 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.157877 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nzhfr" Sep 09 17:18:48.218529 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.218506 2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.231997 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-conf-dir\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232026 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-etc-kubernetes\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232050 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/095893a3-d652-49a4-b6ae-aa6e67db5938-agent-certs\") pod \"konnectivity-agent-q7ctz\" (UID: \"095893a3-d652-49a4-b6ae-aa6e67db5938\") " pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232074 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ck2\" (UniqueName: \"kubernetes.io/projected/fc030ff5-217f-41e5-8b48-bc45036de749-kube-api-access-c7ck2\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232103 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-host\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232134 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-tuned\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232152 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a5ff466-6997-438f-8364-7277a690051d-serviceca\") pod \"node-ca-tt8b9\" (UID: \"2a5ff466-6997-438f-8364-7277a690051d\") " pod="openshift-image-registry/node-ca-tt8b9" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232165 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-cnibin\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232179 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-sysctl-conf\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232193 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnk2q\" (UniqueName: \"kubernetes.io/projected/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-kube-api-access-gnk2q\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232208 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/690ed990-fef1-46c2-b214-3d2ba5fb78c4-env-overrides\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232223 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-modprobe-d\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232243 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-kubernetes\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232266 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc030ff5-217f-41e5-8b48-bc45036de749-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232290 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-var-lib-cni-multus\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232313 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1028b34-ff23-48f2-b853-0b0cdf0fe595-host-slash\") pod \"iptables-alerter-xhtvk\" (UID: \"b1028b34-ff23-48f2-b853-0b0cdf0fe595\") " pod="openshift-network-operator/iptables-alerter-xhtvk" Sep 09 17:18:48.233377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232336 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-cni-netd\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232360 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-sysconfig\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232380 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-sys\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232411 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/22945efa-5c74-4bb9-b708-41a486a7e7c8-dbus\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232437 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-registration-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232467 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-cni-dir\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232490 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-node-log\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232513 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/690ed990-fef1-46c2-b214-3d2ba5fb78c4-ovn-node-metrics-cert\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232543 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh8zt\" (UniqueName: \"kubernetes.io/projected/2a5ff466-6997-438f-8364-7277a690051d-kube-api-access-rh8zt\") pod \"node-ca-tt8b9\" (UID: \"2a5ff466-6997-438f-8364-7277a690051d\") " pod="openshift-image-registry/node-ca-tt8b9" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232567 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc030ff5-217f-41e5-8b48-bc45036de749-cni-binary-copy\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232591 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-run-multus-certs\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232636 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zh4c\" (UniqueName: \"kubernetes.io/projected/c137012e-8c0e-4444-8c6b-f001185bfc4e-kube-api-access-2zh4c\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232680 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-systemd-units\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232705 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbchp\" (UniqueName: \"kubernetes.io/projected/b88c2732-169f-47a7-b117-b556624997e7-kube-api-access-pbchp\") pod \"node-resolver-54x4s\" (UID: \"b88c2732-169f-47a7-b117-b556624997e7\") " pod="openshift-dns/node-resolver-54x4s" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232730 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232755 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-device-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232777 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-daemon-config\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.234182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232801 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/690ed990-fef1-46c2-b214-3d2ba5fb78c4-ovnkube-config\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232827 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/690ed990-fef1-46c2-b214-3d2ba5fb78c4-ovnkube-script-lib\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232850 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-tmp\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232871 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-run-netns\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232897 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp4fx\" (UniqueName: \"kubernetes.io/projected/b1028b34-ff23-48f2-b853-0b0cdf0fe595-kube-api-access-jp4fx\") pod \"iptables-alerter-xhtvk\" (UID: \"b1028b34-ff23-48f2-b853-0b0cdf0fe595\") " pod="openshift-network-operator/iptables-alerter-xhtvk" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232912 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a5ff466-6997-438f-8364-7277a690051d-host\") pod \"node-ca-tt8b9\" (UID: \"2a5ff466-6997-438f-8364-7277a690051d\") " pod="openshift-image-registry/node-ca-tt8b9" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232926 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc030ff5-217f-41e5-8b48-bc45036de749-cnibin\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232942 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4qn4\" (UniqueName: \"kubernetes.io/projected/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-kube-api-access-t4qn4\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.232957 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8dqf\" (UniqueName: \"kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf\") pod \"network-check-target-f6l6s\" (UID: \"7dc76395-0bd2-439e-b9f9-2c89105cc455\") " pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233004 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-kubelet\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233030 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-etc-openvswitch\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233068 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-run-ovn\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233103 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-host\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233161 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-systemd-units\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233194 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-sysconfig\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233229 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-cni-netd\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233231 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-run-ovn\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.234900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233269 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a5ff466-6997-438f-8364-7277a690051d-host\") pod \"node-ca-tt8b9\" (UID: \"2a5ff466-6997-438f-8364-7277a690051d\") " pod="openshift-image-registry/node-ca-tt8b9" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233283 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-sysctl-conf\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233288 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-kubelet\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233314 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-etc-openvswitch\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233336 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-sys\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233792 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-node-log\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233894 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a5ff466-6997-438f-8364-7277a690051d-serviceca\") pod \"node-ca-tt8b9\" (UID: \"2a5ff466-6997-438f-8364-7277a690051d\") " pod="openshift-image-registry/node-ca-tt8b9" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233936 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-modprobe-d\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233942 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1028b34-ff23-48f2-b853-0b0cdf0fe595-host-slash\") pod \"iptables-alerter-xhtvk\" (UID: \"b1028b34-ff23-48f2-b853-0b0cdf0fe595\") " pod="openshift-network-operator/iptables-alerter-xhtvk" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.233994 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234037 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234041 2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234065 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-var-lib-cni-bin\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234091 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-run\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234124 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234139 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-run\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234180 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-lib-modules\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234191 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-kubernetes\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.235597 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234218 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-socket-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234247 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-socket-dir-parent\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234285 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-cni-bin\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234311 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234336 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-var-lib-kubelet\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234336 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-lib-modules\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234389 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234395 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-cni-bin\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234436 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1028b34-ff23-48f2-b853-0b0cdf0fe595-iptables-alerter-script\") pod \"iptables-alerter-xhtvk\" (UID: \"b1028b34-ff23-48f2-b853-0b0cdf0fe595\") " pod="openshift-network-operator/iptables-alerter-xhtvk" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234486 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxvcq\" (UniqueName: \"kubernetes.io/projected/690ed990-fef1-46c2-b214-3d2ba5fb78c4-kube-api-access-wxvcq\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234595 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-sysctl-d\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234644 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/095893a3-d652-49a4-b6ae-aa6e67db5938-konnectivity-ca\") pod \"konnectivity-agent-q7ctz\" (UID: \"095893a3-d652-49a4-b6ae-aa6e67db5938\") " pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234684 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-run-k8s-cni-cncf-io\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234708 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-hostroot\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234735 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-log-socket\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234760 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-systemd\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234799 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc030ff5-217f-41e5-8b48-bc45036de749-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.236397 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234824 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-sys-fs\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234853 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-var-lib-openvswitch\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234890 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/690ed990-fef1-46c2-b214-3d2ba5fb78c4-env-overrides\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234898 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-run-openvswitch\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234922 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-var-lib-kubelet\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234950 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-systemd\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.234985 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b88c2732-169f-47a7-b117-b556624997e7-hosts-file\") pod \"node-resolver-54x4s\" (UID: \"b88c2732-169f-47a7-b117-b556624997e7\") " pod="openshift-dns/node-resolver-54x4s" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235010 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88c2732-169f-47a7-b117-b556624997e7-tmp-dir\") pod \"node-resolver-54x4s\" (UID: \"b88c2732-169f-47a7-b117-b556624997e7\") " pod="openshift-dns/node-resolver-54x4s" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235015 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1028b34-ff23-48f2-b853-0b0cdf0fe595-iptables-alerter-script\") pod \"iptables-alerter-xhtvk\" (UID: \"b1028b34-ff23-48f2-b853-0b0cdf0fe595\") " pod="openshift-network-operator/iptables-alerter-xhtvk" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235042 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-run-openvswitch\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235044 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/22945efa-5c74-4bb9-b708-41a486a7e7c8-kubelet-config\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235080 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c137012e-8c0e-4444-8c6b-f001185bfc4e-cni-binary-copy\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235108 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-run-netns\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235106 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b88c2732-169f-47a7-b117-b556624997e7-hosts-file\") pod \"node-resolver-54x4s\" (UID: \"b88c2732-169f-47a7-b117-b556624997e7\") " pod="openshift-dns/node-resolver-54x4s" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235157 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-run-systemd\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235163 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-run-netns\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235187 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.237213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235222 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-slash\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235259 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-sysctl-d\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235267 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-var-lib-kubelet\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235298 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88c2732-169f-47a7-b117-b556624997e7-tmp-dir\") pod \"node-resolver-54x4s\" (UID: \"b88c2732-169f-47a7-b117-b556624997e7\") " pod="openshift-dns/node-resolver-54x4s" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235305 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc030ff5-217f-41e5-8b48-bc45036de749-system-cni-dir\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235319 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-var-lib-openvswitch\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235346 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc030ff5-217f-41e5-8b48-bc45036de749-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235361 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-log-socket\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235368 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-run-systemd\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235387 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-slash\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235410 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/690ed990-fef1-46c2-b214-3d2ba5fb78c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235412 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wjl\" (UniqueName: \"kubernetes.io/projected/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-kube-api-access-g5wjl\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235451 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-system-cni-dir\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.235889 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/690ed990-fef1-46c2-b214-3d2ba5fb78c4-ovnkube-config\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.238012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.236163 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/690ed990-fef1-46c2-b214-3d2ba5fb78c4-ovnkube-script-lib\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.240374 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.239129 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-etc-tuned\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.240374 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.240114 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-tmp\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.241029 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.241009 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/690ed990-fef1-46c2-b214-3d2ba5fb78c4-ovn-node-metrics-cert\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.246525 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.246451 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbchp\" (UniqueName: \"kubernetes.io/projected/b88c2732-169f-47a7-b117-b556624997e7-kube-api-access-pbchp\") pod \"node-resolver-54x4s\" (UID: \"b88c2732-169f-47a7-b117-b556624997e7\") " pod="openshift-dns/node-resolver-54x4s" Sep 09 17:18:48.246666 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.246645 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnk2q\" (UniqueName: \"kubernetes.io/projected/16414bac-4ae6-4cb6-bb88-b1fe2906b3c6-kube-api-access-gnk2q\") pod \"tuned-rvdl4\" (UID: \"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6\") " pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.247402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.247382 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp4fx\" (UniqueName: \"kubernetes.io/projected/b1028b34-ff23-48f2-b853-0b0cdf0fe595-kube-api-access-jp4fx\") pod \"iptables-alerter-xhtvk\" (UID: \"b1028b34-ff23-48f2-b853-0b0cdf0fe595\") " pod="openshift-network-operator/iptables-alerter-xhtvk" Sep 09 17:18:48.247936 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.247831 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh8zt\" (UniqueName: \"kubernetes.io/projected/2a5ff466-6997-438f-8364-7277a690051d-kube-api-access-rh8zt\") pod \"node-ca-tt8b9\" (UID: \"2a5ff466-6997-438f-8364-7277a690051d\") " pod="openshift-image-registry/node-ca-tt8b9" Sep 09 17:18:48.247936 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.247901 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxvcq\" (UniqueName: \"kubernetes.io/projected/690ed990-fef1-46c2-b214-3d2ba5fb78c4-kube-api-access-wxvcq\") pod \"ovnkube-node-xkdmr\" (UID: \"690ed990-fef1-46c2-b214-3d2ba5fb78c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.336232 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336193 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc030ff5-217f-41e5-8b48-bc45036de749-system-cni-dir\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.336399 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336246 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc030ff5-217f-41e5-8b48-bc45036de749-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.336399 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336276 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wjl\" (UniqueName: \"kubernetes.io/projected/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-kube-api-access-g5wjl\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.336399 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336281 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc030ff5-217f-41e5-8b48-bc45036de749-system-cni-dir\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.336399 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336300 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-system-cni-dir\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.336399 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336327 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-conf-dir\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.336399 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336381 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-conf-dir\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.336699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336407 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-system-cni-dir\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.336699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336451 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-etc-kubernetes\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.336699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336482 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/095893a3-d652-49a4-b6ae-aa6e67db5938-agent-certs\") pod \"konnectivity-agent-q7ctz\" (UID: \"095893a3-d652-49a4-b6ae-aa6e67db5938\") " pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:18:48.336699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336510 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ck2\" (UniqueName: \"kubernetes.io/projected/fc030ff5-217f-41e5-8b48-bc45036de749-kube-api-access-c7ck2\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.336699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336541 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-cnibin\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.336699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336553 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-etc-kubernetes\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.336699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336568 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc030ff5-217f-41e5-8b48-bc45036de749-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.336699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336613 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-var-lib-cni-multus\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.336699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336660 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/22945efa-5c74-4bb9-b708-41a486a7e7c8-dbus\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:48.336699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336683 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-registration-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336703 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-cni-dir\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336731 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc030ff5-217f-41e5-8b48-bc45036de749-cni-binary-copy\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336754 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-run-multus-certs\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336777 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zh4c\" (UniqueName: \"kubernetes.io/projected/c137012e-8c0e-4444-8c6b-f001185bfc4e-kube-api-access-2zh4c\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336806 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336830 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-device-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336854 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-daemon-config\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336883 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-run-netns\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336897 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc030ff5-217f-41e5-8b48-bc45036de749-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336909 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc030ff5-217f-41e5-8b48-bc45036de749-cnibin\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336917 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/22945efa-5c74-4bb9-b708-41a486a7e7c8-dbus\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336933 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qn4\" (UniqueName: \"kubernetes.io/projected/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-kube-api-access-t4qn4\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336959 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-registration-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.336964 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dqf\" (UniqueName: \"kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf\") pod \"network-check-target-f6l6s\" (UID: \"7dc76395-0bd2-439e-b9f9-2c89105cc455\") " pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337004 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-cnibin\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337024 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.337115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337052 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-var-lib-cni-bin\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337079 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-socket-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337098 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-run-netns\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337105 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-socket-dir-parent\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337137 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337179 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-cni-dir\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337209 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-var-lib-kubelet\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337226 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-device-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337238 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337262 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337270 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-var-lib-cni-bin\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337267 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/095893a3-d652-49a4-b6ae-aa6e67db5938-konnectivity-ca\") pod \"konnectivity-agent-q7ctz\" (UID: \"095893a3-d652-49a4-b6ae-aa6e67db5938\") " pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337304 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-run-k8s-cni-cncf-io\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337309 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc030ff5-217f-41e5-8b48-bc45036de749-cnibin\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.337100 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337321 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-socket-dir-parent\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337362 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-run-k8s-cni-cncf-io\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337374 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-hostroot\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.337957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337366 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-socket-dir\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337405 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c137012e-8c0e-4444-8c6b-f001185bfc4e-multus-daemon-config\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337417 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-run-multus-certs\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.337421 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret podName:22945efa-5c74-4bb9-b708-41a486a7e7c8 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:48.837401394 +0000 UTC m=+3.077307742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret") pod "global-pull-secret-syncer-zlhb4" (UID: "22945efa-5c74-4bb9-b708-41a486a7e7c8") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337328 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-hostroot\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337443 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-var-lib-cni-multus\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337469 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc030ff5-217f-41e5-8b48-bc45036de749-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337491 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337496 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-sys-fs\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337529 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/22945efa-5c74-4bb9-b708-41a486a7e7c8-kubelet-config\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337537 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c137012e-8c0e-4444-8c6b-f001185bfc4e-host-var-lib-kubelet\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337552 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c137012e-8c0e-4444-8c6b-f001185bfc4e-cni-binary-copy\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.337641 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.337700 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs podName:8a9f88fb-c187-4672-bd8e-1916d5f9ff54 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:48.837682365 +0000 UTC m=+3.077588717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs") pod "network-metrics-daemon-tc2sl" (UID: "8a9f88fb-c187-4672-bd8e-1916d5f9ff54") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337762 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-sys-fs\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337787 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc030ff5-217f-41e5-8b48-bc45036de749-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.338797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337828 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/22945efa-5c74-4bb9-b708-41a486a7e7c8-kubelet-config\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:48.339429 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337887 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/095893a3-d652-49a4-b6ae-aa6e67db5938-konnectivity-ca\") pod \"konnectivity-agent-q7ctz\" (UID: \"095893a3-d652-49a4-b6ae-aa6e67db5938\") " pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:18:48.339429 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.337928 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc030ff5-217f-41e5-8b48-bc45036de749-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.339429 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.338019 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc030ff5-217f-41e5-8b48-bc45036de749-cni-binary-copy\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.339429 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.338055 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c137012e-8c0e-4444-8c6b-f001185bfc4e-cni-binary-copy\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.339875 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.339848 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/095893a3-d652-49a4-b6ae-aa6e67db5938-agent-certs\") pod \"konnectivity-agent-q7ctz\" (UID: \"095893a3-d652-49a4-b6ae-aa6e67db5938\") " pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:18:48.343106 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.343086 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:18:48.343195 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.343111 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:18:48.343195 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.343124 2553 projected.go:196] Error preparing data for projected volume kube-api-access-k8dqf for pod openshift-network-diagnostics/network-check-target-f6l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:18:48.343335 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.343238 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf podName:7dc76395-0bd2-439e-b9f9-2c89105cc455 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:48.843214118 +0000 UTC m=+3.083120469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k8dqf" (UniqueName: "kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf") pod "network-check-target-f6l6s" (UID: "7dc76395-0bd2-439e-b9f9-2c89105cc455") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:18:48.345673 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.345550 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ck2\" (UniqueName: \"kubernetes.io/projected/fc030ff5-217f-41e5-8b48-bc45036de749-kube-api-access-c7ck2\") pod \"multus-additional-cni-plugins-wdsct\" (UID: \"fc030ff5-217f-41e5-8b48-bc45036de749\") " pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.346834 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.346532 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qn4\" (UniqueName: \"kubernetes.io/projected/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-kube-api-access-t4qn4\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:48.347099 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.347078 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zh4c\" (UniqueName: \"kubernetes.io/projected/c137012e-8c0e-4444-8c6b-f001185bfc4e-kube-api-access-2zh4c\") pod \"multus-zg2bq\" (UID: \"c137012e-8c0e-4444-8c6b-f001185bfc4e\") " pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.347183 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.347117 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wjl\" (UniqueName: \"kubernetes.io/projected/a81b68e0-f5dc-4c38-90cb-e8b9e2939af9-kube-api-access-g5wjl\") pod \"aws-ebs-csi-driver-node-8hb78\" (UID: \"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.349227 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.349144 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" event={"ID":"39af5840e15752b8ef00c7c30eaa9a5a","Type":"ContainerStarted","Data":"fc9e1117a71f19b2e2906c2aa285a7d89f43468dbe8e8ce8eea1e831e00d0b6b"} Sep 09 17:18:48.350819 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.350789 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" event={"ID":"0a288a822be8e393a56ffcbb08fdf962","Type":"ContainerStarted","Data":"4b5ca4a3765572c92f8ff475591987fffbc639db3550371d99058d58ddf3ffa4"} Sep 09 17:18:48.427434 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.427405 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xhtvk" Sep 09 17:18:48.442675 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.442641 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:18:48.452464 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.452443 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" Sep 09 17:18:48.458366 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.458344 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-54x4s" Sep 09 17:18:48.465966 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.465943 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tt8b9" Sep 09 17:18:48.472840 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.472818 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:18:48.480374 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.480356 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wdsct" Sep 09 17:18:48.489861 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.489841 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" Sep 09 17:18:48.494771 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.494749 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zg2bq" Sep 09 17:18:48.611793 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.611717 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 17:18:48.841234 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.841198 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:48.841393 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.841268 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:48.841393 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.841360 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:18:48.841484 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.841429 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs podName:8a9f88fb-c187-4672-bd8e-1916d5f9ff54 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:49.841408031 +0000 UTC m=+4.081314376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs") pod "network-metrics-daemon-tc2sl" (UID: "8a9f88fb-c187-4672-bd8e-1916d5f9ff54") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:18:48.841484 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.841362 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:18:48.841484 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.841478 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret podName:22945efa-5c74-4bb9-b708-41a486a7e7c8 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:49.841467892 +0000 UTC m=+4.081374236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret") pod "global-pull-secret-syncer-zlhb4" (UID: "22945efa-5c74-4bb9-b708-41a486a7e7c8") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:18:48.942019 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:48.941985 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dqf\" (UniqueName: \"kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf\") pod \"network-check-target-f6l6s\" (UID: \"7dc76395-0bd2-439e-b9f9-2c89105cc455\") " pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:48.942213 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.942144 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:18:48.942213 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.942173 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:18:48.942213 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.942186 2553 projected.go:196] Error preparing data for projected volume kube-api-access-k8dqf for pod openshift-network-diagnostics/network-check-target-f6l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:18:48.942347 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:48.942252 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf podName:7dc76395-0bd2-439e-b9f9-2c89105cc455 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:49.942233284 +0000 UTC m=+4.182139632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8dqf" (UniqueName: "kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf") pod "network-check-target-f6l6s" (UID: "7dc76395-0bd2-439e-b9f9-2c89105cc455") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:18:49.160180 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:49.160134 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 17:13:48 +0000 UTC" deadline="2028-03-31 11:17:42.383607084 +0000 UTC" Sep 09 17:18:49.160180 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:49.160175 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13649h58m53.223437449s" Sep 09 17:18:49.342370 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:49.342299 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:49.342537 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:49.342307 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:49.342537 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:49.342433 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:18:49.342537 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:49.342466 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:18:49.370590 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:49.370563 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc137012e_8c0e_4444_8c6b_f001185bfc4e.slice/crio-4ba9e66a73f415bcaaeb3c41e8410faa9bab5ad9b1dcec3a2263a0ce2f3dbcd2 WatchSource:0}: Error finding container 4ba9e66a73f415bcaaeb3c41e8410faa9bab5ad9b1dcec3a2263a0ce2f3dbcd2: Status 404 returned error can't find the container with id 4ba9e66a73f415bcaaeb3c41e8410faa9bab5ad9b1dcec3a2263a0ce2f3dbcd2 Sep 09 17:18:49.372827 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:49.372808 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690ed990_fef1_46c2_b214_3d2ba5fb78c4.slice/crio-37540ce723af1b3c41885dc69b7cbe2ff37df4c14a709a4eec88e3fe5b458696 WatchSource:0}: Error finding container 37540ce723af1b3c41885dc69b7cbe2ff37df4c14a709a4eec88e3fe5b458696: Status 404 returned error can't find the container with id 37540ce723af1b3c41885dc69b7cbe2ff37df4c14a709a4eec88e3fe5b458696 Sep 09 17:18:49.373910 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:49.373790 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095893a3_d652_49a4_b6ae_aa6e67db5938.slice/crio-96cefac080913e78997e936ec8976be7d7beb58936b7aa431c0dbdd1e1a965d8 WatchSource:0}: Error finding container 96cefac080913e78997e936ec8976be7d7beb58936b7aa431c0dbdd1e1a965d8: Status 404 returned error can't find the container with id 96cefac080913e78997e936ec8976be7d7beb58936b7aa431c0dbdd1e1a965d8 Sep 09 17:18:49.376962 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:49.376940 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5ff466_6997_438f_8364_7277a690051d.slice/crio-d870c1018e25cfe76baddd481f880b693dd424b0ffe111372676a3274106be34 WatchSource:0}: Error finding container d870c1018e25cfe76baddd481f880b693dd424b0ffe111372676a3274106be34: Status 404 returned error can't find the container with id d870c1018e25cfe76baddd481f880b693dd424b0ffe111372676a3274106be34 Sep 09 17:18:49.377388 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:49.377370 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16414bac_4ae6_4cb6_bb88_b1fe2906b3c6.slice/crio-c973785a643925619e94b27cbdbd51bf3b1a820b806231b4d5f63d72e975abdd WatchSource:0}: Error finding container c973785a643925619e94b27cbdbd51bf3b1a820b806231b4d5f63d72e975abdd: Status 404 returned error can't find the container with id c973785a643925619e94b27cbdbd51bf3b1a820b806231b4d5f63d72e975abdd Sep 09 17:18:49.379423 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:49.379403 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc030ff5_217f_41e5_8b48_bc45036de749.slice/crio-afe829f091fb118e88f8b56af766786badace37b2855dcf10c31a85958ec489d WatchSource:0}: Error finding container afe829f091fb118e88f8b56af766786badace37b2855dcf10c31a85958ec489d: Status 404 returned error can't find the container with id afe829f091fb118e88f8b56af766786badace37b2855dcf10c31a85958ec489d Sep 09 17:18:49.379928 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:18:49.379809 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1028b34_ff23_48f2_b853_0b0cdf0fe595.slice/crio-d933876291be49383a0e1bda80c9f4e88bb183a0fdb0c6aa56440ba6aca754ab WatchSource:0}: Error finding container d933876291be49383a0e1bda80c9f4e88bb183a0fdb0c6aa56440ba6aca754ab: Status 404 returned error can't find the container with id d933876291be49383a0e1bda80c9f4e88bb183a0fdb0c6aa56440ba6aca754ab Sep 09 17:18:49.848542 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:49.848051 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:49.848542 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:49.848138 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:49.848542 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:49.848284 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:18:49.848542 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:49.848345 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret podName:22945efa-5c74-4bb9-b708-41a486a7e7c8 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:51.848326468 +0000 UTC m=+6.088232811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret") pod "global-pull-secret-syncer-zlhb4" (UID: "22945efa-5c74-4bb9-b708-41a486a7e7c8") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:18:49.848542 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:49.848413 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:18:49.848542 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:49.848489 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs podName:8a9f88fb-c187-4672-bd8e-1916d5f9ff54 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:51.848469142 +0000 UTC m=+6.088375492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs") pod "network-metrics-daemon-tc2sl" (UID: "8a9f88fb-c187-4672-bd8e-1916d5f9ff54") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:18:49.949342 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:49.949299 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dqf\" (UniqueName: \"kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf\") pod \"network-check-target-f6l6s\" (UID: \"7dc76395-0bd2-439e-b9f9-2c89105cc455\") " pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:49.949548 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:49.949530 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:18:49.949651 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:49.949558 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:18:49.949651 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:49.949571 2553 projected.go:196] Error preparing data for projected volume kube-api-access-k8dqf for pod openshift-network-diagnostics/network-check-target-f6l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:18:49.949766 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:49.949680 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf podName:7dc76395-0bd2-439e-b9f9-2c89105cc455 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:51.949661808 +0000 UTC m=+6.189568152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8dqf" (UniqueName: "kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf") pod "network-check-target-f6l6s" (UID: "7dc76395-0bd2-439e-b9f9-2c89105cc455") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:18:50.160611 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.160518 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 17:13:48 +0000 UTC" deadline="2028-04-13 05:18:09.27002619 +0000 UTC" Sep 09 17:18:50.160611 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.160578 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13955h59m19.109451956s" Sep 09 17:18:50.345228 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.344580 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:50.345228 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:50.344789 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:18:50.366235 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.363106 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-54x4s" event={"ID":"b88c2732-169f-47a7-b117-b556624997e7","Type":"ContainerStarted","Data":"1d030ef84fb564bb6046a15fbdb38205717c79907253f7fbc5a688a3994c4559"} Sep 09 17:18:50.366235 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.365995 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xhtvk" event={"ID":"b1028b34-ff23-48f2-b853-0b0cdf0fe595","Type":"ContainerStarted","Data":"d933876291be49383a0e1bda80c9f4e88bb183a0fdb0c6aa56440ba6aca754ab"} Sep 09 17:18:50.368395 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.368368 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tt8b9" event={"ID":"2a5ff466-6997-438f-8364-7277a690051d","Type":"ContainerStarted","Data":"d870c1018e25cfe76baddd481f880b693dd424b0ffe111372676a3274106be34"} Sep 09 17:18:50.375560 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.375447 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" event={"ID":"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6","Type":"ContainerStarted","Data":"c973785a643925619e94b27cbdbd51bf3b1a820b806231b4d5f63d72e975abdd"} Sep 09 17:18:50.381649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.381613 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdsct" event={"ID":"fc030ff5-217f-41e5-8b48-bc45036de749","Type":"ContainerStarted","Data":"afe829f091fb118e88f8b56af766786badace37b2855dcf10c31a85958ec489d"} Sep 09 17:18:50.386348 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.386241 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" event={"ID":"690ed990-fef1-46c2-b214-3d2ba5fb78c4","Type":"ContainerStarted","Data":"37540ce723af1b3c41885dc69b7cbe2ff37df4c14a709a4eec88e3fe5b458696"} Sep 09 17:18:50.391111 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.391051 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q7ctz" event={"ID":"095893a3-d652-49a4-b6ae-aa6e67db5938","Type":"ContainerStarted","Data":"96cefac080913e78997e936ec8976be7d7beb58936b7aa431c0dbdd1e1a965d8"} Sep 09 17:18:50.397651 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.397611 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zg2bq" event={"ID":"c137012e-8c0e-4444-8c6b-f001185bfc4e","Type":"ContainerStarted","Data":"4ba9e66a73f415bcaaeb3c41e8410faa9bab5ad9b1dcec3a2263a0ce2f3dbcd2"} Sep 09 17:18:50.424178 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.424144 2553 generic.go:356] "Generic (PLEG): container finished" podID="39af5840e15752b8ef00c7c30eaa9a5a" containerID="f1b7c5fad004cabef74d5c7b12e23571b9613562b40b2da6762b40adaed6b894" exitCode=0 Sep 09 17:18:50.424823 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.424798 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" event={"ID":"39af5840e15752b8ef00c7c30eaa9a5a","Type":"ContainerDied","Data":"f1b7c5fad004cabef74d5c7b12e23571b9613562b40b2da6762b40adaed6b894"} Sep 09 17:18:50.433744 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.433686 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" event={"ID":"0a288a822be8e393a56ffcbb08fdf962","Type":"ContainerStarted","Data":"7f8bcf91f5fe141f6abd30f9d1f6a4c247f7af2f248aa0b072b5af0d7ba3c166"} Sep 09 17:18:50.447170 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:50.447086 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" event={"ID":"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9","Type":"ContainerStarted","Data":"4602729e97a75a07954e28c34b42302ad1adf4fd2486e0cdac57ed2db0cc6785"} Sep 09 17:18:51.342022 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:51.341990 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:51.342482 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:51.342117 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:18:51.342542 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:51.342522 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:51.342684 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:51.342637 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:18:51.463277 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:51.463241 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" event={"ID":"39af5840e15752b8ef00c7c30eaa9a5a","Type":"ContainerStarted","Data":"7e5c641db05939431f176b422cb57e9655da93f2bfc09294a09a3a372b344ce6"} Sep 09 17:18:51.479076 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:51.478973 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-213.ec2.internal" podStartSLOduration=4.478954678 podStartE2EDuration="4.478954678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:18:50.457003125 +0000 UTC m=+4.696909480" watchObservedRunningTime="2026-09-09 17:18:51.478954678 +0000 UTC m=+5.718861032" Sep 09 17:18:51.479431 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:51.479395 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-213.ec2.internal" podStartSLOduration=4.479382344 podStartE2EDuration="4.479382344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:18:51.478347381 +0000 UTC m=+5.718253737" watchObservedRunningTime="2026-09-09 17:18:51.479382344 +0000 UTC m=+5.719288698" Sep 09 17:18:51.864610 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:51.864527 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:51.864775 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:51.864615 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:51.864847 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:51.864778 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:18:51.864902 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:51.864846 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret podName:22945efa-5c74-4bb9-b708-41a486a7e7c8 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:55.864828055 +0000 UTC m=+10.104734405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret") pod "global-pull-secret-syncer-zlhb4" (UID: "22945efa-5c74-4bb9-b708-41a486a7e7c8") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:18:51.865279 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:51.865258 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:18:51.865361 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:51.865319 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs podName:8a9f88fb-c187-4672-bd8e-1916d5f9ff54 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:55.86530269 +0000 UTC m=+10.105209034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs") pod "network-metrics-daemon-tc2sl" (UID: "8a9f88fb-c187-4672-bd8e-1916d5f9ff54") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:18:51.966035 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:51.965435 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dqf\" (UniqueName: \"kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf\") pod \"network-check-target-f6l6s\" (UID: \"7dc76395-0bd2-439e-b9f9-2c89105cc455\") " pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:51.966035 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:51.965610 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:18:51.966035 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:51.965645 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:18:51.966035 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:51.965659 2553 projected.go:196] Error preparing data for projected volume kube-api-access-k8dqf for pod openshift-network-diagnostics/network-check-target-f6l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:18:51.966035 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:51.965716 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf podName:7dc76395-0bd2-439e-b9f9-2c89105cc455 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:55.965699245 +0000 UTC m=+10.205605589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8dqf" (UniqueName: "kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf") pod "network-check-target-f6l6s" (UID: "7dc76395-0bd2-439e-b9f9-2c89105cc455") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:18:52.341832 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:52.341801 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:52.342021 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:52.341914 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:18:53.342508 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:53.342461 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:53.342508 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:53.342478 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:53.343015 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:53.342561 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:18:53.343015 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:53.342712 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:18:54.343336 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:54.343301 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:54.343823 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:54.343442 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:18:55.342528 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:55.342285 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:55.342720 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:55.342285 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:55.342720 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:55.342662 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:18:55.342830 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:55.342730 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:18:55.894329 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:55.893698 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:55.894329 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:55.893775 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:55.894329 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:55.893931 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:18:55.894329 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:55.893996 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs podName:8a9f88fb-c187-4672-bd8e-1916d5f9ff54 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:03.893976352 +0000 UTC m=+18.133882704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs") pod "network-metrics-daemon-tc2sl" (UID: "8a9f88fb-c187-4672-bd8e-1916d5f9ff54") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:18:55.894329 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:55.894268 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:18:55.894329 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:55.894326 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret podName:22945efa-5c74-4bb9-b708-41a486a7e7c8 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:03.894309744 +0000 UTC m=+18.134216080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret") pod "global-pull-secret-syncer-zlhb4" (UID: "22945efa-5c74-4bb9-b708-41a486a7e7c8") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:18:55.994148 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:55.994110 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dqf\" (UniqueName: \"kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf\") pod \"network-check-target-f6l6s\" (UID: \"7dc76395-0bd2-439e-b9f9-2c89105cc455\") " pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:55.994368 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:55.994329 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:18:55.994368 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:55.994348 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:18:55.994368 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:55.994361 2553 projected.go:196] Error preparing data for projected volume kube-api-access-k8dqf for pod openshift-network-diagnostics/network-check-target-f6l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:18:55.994530 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:55.994419 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf podName:7dc76395-0bd2-439e-b9f9-2c89105cc455 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:03.994402144 +0000 UTC m=+18.234308480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8dqf" (UniqueName: "kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf") pod "network-check-target-f6l6s" (UID: "7dc76395-0bd2-439e-b9f9-2c89105cc455") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:18:56.342723 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:56.342690 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:56.342891 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:56.342815 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:18:57.341813 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:57.341782 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:57.342261 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:57.341825 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:57.342261 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:57.341911 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:18:57.342261 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:57.342018 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:18:58.342453 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:58.342422 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:18:58.342901 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:58.342555 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:18:59.341683 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:59.341648 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:18:59.341859 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:18:59.341648 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:18:59.341859 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:59.341782 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:18:59.341990 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:18:59.341857 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:00.342491 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:00.342453 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:00.342936 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:00.342603 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:01.341908 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:01.341874 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:01.341908 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:01.341911 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:01.342139 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:01.342014 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:01.342139 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:01.342107 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:02.342077 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:02.342038 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:02.342552 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:02.342188 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:03.342383 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:03.342355 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:03.342383 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:03.342379 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:03.342899 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:03.342472 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:03.342899 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:03.342640 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:03.949978 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:03.949948 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:03.950167 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:03.950022 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:03.950167 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:03.950118 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:19:03.950295 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:03.950195 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:19:03.950295 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:03.950200 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret podName:22945efa-5c74-4bb9-b708-41a486a7e7c8 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:19.950177636 +0000 UTC m=+34.190083971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret") pod "global-pull-secret-syncer-zlhb4" (UID: "22945efa-5c74-4bb9-b708-41a486a7e7c8") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:19:03.950295 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:03.950273 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs podName:8a9f88fb-c187-4672-bd8e-1916d5f9ff54 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:19.950255171 +0000 UTC m=+34.190161518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs") pod "network-metrics-daemon-tc2sl" (UID: "8a9f88fb-c187-4672-bd8e-1916d5f9ff54") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:19:04.050905 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:04.050870 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dqf\" (UniqueName: \"kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf\") pod \"network-check-target-f6l6s\" (UID: \"7dc76395-0bd2-439e-b9f9-2c89105cc455\") " pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:04.051097 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:04.051033 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:19:04.051097 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:04.051054 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:19:04.051097 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:04.051064 2553 projected.go:196] Error preparing data for projected volume kube-api-access-k8dqf for pod openshift-network-diagnostics/network-check-target-f6l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:19:04.051240 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:04.051121 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf podName:7dc76395-0bd2-439e-b9f9-2c89105cc455 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:20.051102122 +0000 UTC m=+34.291008477 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8dqf" (UniqueName: "kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf") pod "network-check-target-f6l6s" (UID: "7dc76395-0bd2-439e-b9f9-2c89105cc455") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:19:04.341973 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:04.341891 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:04.342131 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:04.342031 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:05.342225 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:05.342196 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:05.342225 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:05.342197 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:05.342673 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:05.342324 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:05.342673 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:05.342377 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:06.342545 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:06.342509 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:06.342909 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:06.342660 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:07.341745 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.341513 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:07.341745 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:07.341603 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:07.341745 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.341645 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:07.341745 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:07.341714 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:07.496940 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.496713 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" event={"ID":"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9","Type":"ContainerStarted","Data":"fb4a2adba2d5a4cbef093ab7a610f424a280302a1666379494e6b72a6710c5fe"} Sep 09 17:19:07.498902 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.498852 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-54x4s" event={"ID":"b88c2732-169f-47a7-b117-b556624997e7","Type":"ContainerStarted","Data":"bc135710d0d4a03adb16420db348a5507644f03cef8b089e7ebd1968dafb40e2"} Sep 09 17:19:07.500753 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.500704 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tt8b9" event={"ID":"2a5ff466-6997-438f-8364-7277a690051d","Type":"ContainerStarted","Data":"20803082019fce5ca32bf4811e9fabfd4a1fe2a41d81e45276a9b4f42cde01d2"} Sep 09 17:19:07.503393 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.503345 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" event={"ID":"16414bac-4ae6-4cb6-bb88-b1fe2906b3c6","Type":"ContainerStarted","Data":"2eb3a16f70912943749b8336ad7f0989f2c203399d3021e3af3fba4c34a54c86"} Sep 09 17:19:07.508524 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.508497 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdsct" event={"ID":"fc030ff5-217f-41e5-8b48-bc45036de749","Type":"ContainerStarted","Data":"612a966dcca1d427f7353ffe6b75322072dbf475044bfb51b8ba231fb9a1d546"} Sep 09 17:19:07.513463 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.511869 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" event={"ID":"690ed990-fef1-46c2-b214-3d2ba5fb78c4","Type":"ContainerStarted","Data":"a5f7e98aae84f0779b8b098d050a57ff734e31429cea0e7b4a353d31921068ad"} Sep 09 17:19:07.513463 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.511940 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" event={"ID":"690ed990-fef1-46c2-b214-3d2ba5fb78c4","Type":"ContainerStarted","Data":"dc18d27b18c1375788baeaa6df4efbd8c2c763fd927c28a1403956f2e832aa21"} Sep 09 17:19:07.514833 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.514780 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-54x4s" podStartSLOduration=3.714182537 podStartE2EDuration="21.51476178s" totalImagesPullingTime="17.800579243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:07.513720482 +0000 UTC m=+21.753626836" watchObservedRunningTime="2026-09-09 17:19:07.51476178 +0000 UTC m=+21.754668133" Sep 09 17:19:07.515508 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.515289 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q7ctz" event={"ID":"095893a3-d652-49a4-b6ae-aa6e67db5938","Type":"ContainerStarted","Data":"df9e3b57260149da7e230e509b80f95ae5d41a703526271e223009bd6da4ada9"} Sep 09 17:19:07.516904 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.516879 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zg2bq" event={"ID":"c137012e-8c0e-4444-8c6b-f001185bfc4e","Type":"ContainerStarted","Data":"a1b4d23d663dabb2a332f374ed0bda9bc02b8a867ca60d0c6cb1f65aacac7024"} Sep 09 17:19:07.528756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.528720 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rvdl4" podStartSLOduration=3.722340912 podStartE2EDuration="21.528705661s" totalImagesPullingTime="17.806364749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:07.528535536 +0000 UTC m=+21.768441890" watchObservedRunningTime="2026-09-09 17:19:07.528705661 +0000 UTC m=+21.768612013" Sep 09 17:19:07.559586 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.559545 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tt8b9" podStartSLOduration=3.752953316 podStartE2EDuration="21.559527894s" totalImagesPullingTime="17.806574578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:07.541339407 +0000 UTC m=+21.781245765" watchObservedRunningTime="2026-09-09 17:19:07.559527894 +0000 UTC m=+21.799434225" Sep 09 17:19:07.574213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.574173 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zg2bq" podStartSLOduration=3.697343728 podStartE2EDuration="21.574159468s" totalImagesPullingTime="17.87681574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:07.573798267 +0000 UTC m=+21.813704620" watchObservedRunningTime="2026-09-09 17:19:07.574159468 +0000 UTC m=+21.814065822" Sep 09 17:19:07.589166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:07.589112 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q7ctz" podStartSLOduration=11.210782529 podStartE2EDuration="21.58909288s" totalImagesPullingTime="10.378310351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:07.588932865 +0000 UTC m=+21.828839217" watchObservedRunningTime="2026-09-09 17:19:07.58909288 +0000 UTC m=+21.828999266" Sep 09 17:19:08.342353 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:08.342271 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:08.342542 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:08.342417 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:08.520375 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:08.520342 2553 generic.go:356] "Generic (PLEG): container finished" podID="fc030ff5-217f-41e5-8b48-bc45036de749" containerID="612a966dcca1d427f7353ffe6b75322072dbf475044bfb51b8ba231fb9a1d546" exitCode=0 Sep 09 17:19:08.520882 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:08.520430 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdsct" event={"ID":"fc030ff5-217f-41e5-8b48-bc45036de749","Type":"ContainerDied","Data":"612a966dcca1d427f7353ffe6b75322072dbf475044bfb51b8ba231fb9a1d546"} Sep 09 17:19:08.524335 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:08.524310 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" event={"ID":"690ed990-fef1-46c2-b214-3d2ba5fb78c4","Type":"ContainerStarted","Data":"1bc3eabbe4370f007b11e78f2679e00c2c8f2bf05eaf953cd8f0c3e543432a1c"} Sep 09 17:19:08.524466 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:08.524386 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" event={"ID":"690ed990-fef1-46c2-b214-3d2ba5fb78c4","Type":"ContainerStarted","Data":"457dd2ac20265cd9850766ab6f599a53209da0dd952c207ebad91fd96551a976"} Sep 09 17:19:08.524466 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:08.524434 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" event={"ID":"690ed990-fef1-46c2-b214-3d2ba5fb78c4","Type":"ContainerStarted","Data":"43e4749730e75cef12f0afa5e639b5575bd86c41d884bd987712c33fdb2c6e8d"} Sep 09 17:19:08.524557 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:08.524481 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" event={"ID":"690ed990-fef1-46c2-b214-3d2ba5fb78c4","Type":"ContainerStarted","Data":"a4070bbb34941b51dcfc9f9a7e4927e1efe9fae905cca9d1403d52bc344bf126"} Sep 09 17:19:08.660472 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:08.660255 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:19:08.750984 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:08.750961 2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 17:19:09.330096 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:09.329997 2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T17:19:08.750979571Z","UUID":"97ef8c72-2076-48ed-9a26-2c85fc28a4c6","Handler":null,"Name":"","Endpoint":""} Sep 09 17:19:09.331711 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:09.331687 2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 17:19:09.331827 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:09.331724 2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 17:19:09.342460 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:09.342435 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:09.342572 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:09.342556 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:09.342654 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:09.342611 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:09.342762 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:09.342742 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:09.529355 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:09.529314 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" event={"ID":"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9","Type":"ContainerStarted","Data":"e2cb40f9a2baa280eca7e6675478d20d7d0655490bcd758c9dcc06e14d13d162"} Sep 09 17:19:09.531342 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:09.531312 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xhtvk" event={"ID":"b1028b34-ff23-48f2-b853-0b0cdf0fe595","Type":"ContainerStarted","Data":"26b7bc8db88d813511416524b56853f462a140914ec4e8b05274c9c85bb9dd2e"} Sep 09 17:19:09.548694 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:09.548639 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xhtvk" podStartSLOduration=5.746376243 podStartE2EDuration="23.548602749s" totalImagesPullingTime="17.802226506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:09.548438423 +0000 UTC m=+23.788344776" watchObservedRunningTime="2026-09-09 17:19:09.548602749 +0000 UTC m=+23.788509103" Sep 09 17:19:10.028483 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:10.028457 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:19:10.029275 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:10.029250 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:19:10.342432 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:10.342400 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:10.342580 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:10.342501 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:10.535476 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:10.535440 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" event={"ID":"a81b68e0-f5dc-4c38-90cb-e8b9e2939af9","Type":"ContainerStarted","Data":"50a76ef18c8fb4aae55a9f7bc1b5870ef1f9375b4b7973455462a50d5eaa5e00"} Sep 09 17:19:10.539125 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:10.539097 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" event={"ID":"690ed990-fef1-46c2-b214-3d2ba5fb78c4","Type":"ContainerStarted","Data":"059de5714e2652ae4d80618f834cae0bbf24c5837c491539ae51e47b858c53d9"} Sep 09 17:19:10.539839 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:10.539821 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q7ctz" Sep 09 17:19:10.556030 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:10.555989 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hb78" podStartSLOduration=4.34690646 podStartE2EDuration="24.555974837s" totalImagesPullingTime="20.209068377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:10.555396315 +0000 UTC m=+24.795302668" watchObservedRunningTime="2026-09-09 17:19:10.555974837 +0000 UTC m=+24.795881191" Sep 09 17:19:11.342024 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:11.341998 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:11.342195 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:11.342035 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:11.342195 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:11.342119 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:11.342269 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:11.342230 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:12.350269 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:12.350190 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:12.350668 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:12.350337 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:13.342074 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:13.341919 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:13.342174 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:13.341971 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:13.342174 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:13.342149 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:13.342283 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:13.342245 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:13.547776 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:13.547746 2553 generic.go:356] "Generic (PLEG): container finished" podID="fc030ff5-217f-41e5-8b48-bc45036de749" containerID="51f4dea23b8247c0af0ab7d924c5a0e6195e387df90428fb24de8b152d382e4c" exitCode=0 Sep 09 17:19:13.548341 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:13.547831 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdsct" event={"ID":"fc030ff5-217f-41e5-8b48-bc45036de749","Type":"ContainerDied","Data":"51f4dea23b8247c0af0ab7d924c5a0e6195e387df90428fb24de8b152d382e4c"} Sep 09 17:19:13.551633 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:13.551597 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" event={"ID":"690ed990-fef1-46c2-b214-3d2ba5fb78c4","Type":"ContainerStarted","Data":"e0c0ad0a43c45efcb2e65976132b25954d27de2d682961b68ab4de0b2699cfb1"} Sep 09 17:19:13.551883 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:13.551865 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:19:13.551975 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:13.551937 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:19:13.566146 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:13.566127 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:19:13.596655 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:13.596574 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" podStartSLOduration=9.765250704 podStartE2EDuration="27.596561502s" totalImagesPullingTime="17.831310798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:13.595198779 +0000 UTC m=+27.835105131" watchObservedRunningTime="2026-09-09 17:19:13.596561502 +0000 UTC m=+27.836467854" Sep 09 17:19:14.341934 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:14.341909 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:14.342027 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:14.342008 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:14.555579 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:14.555487 2553 generic.go:356] "Generic (PLEG): container finished" podID="fc030ff5-217f-41e5-8b48-bc45036de749" containerID="0f54cb5601463e672237046e3dcbd9749381a7ddc3ae8d4300119a5ee2df47f0" exitCode=0 Sep 09 17:19:14.556056 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:14.555578 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdsct" event={"ID":"fc030ff5-217f-41e5-8b48-bc45036de749","Type":"ContainerDied","Data":"0f54cb5601463e672237046e3dcbd9749381a7ddc3ae8d4300119a5ee2df47f0"} Sep 09 17:19:14.556187 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:14.556163 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:19:14.570963 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:14.570943 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:19:15.341406 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:15.341374 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:15.341570 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:15.341374 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:15.341570 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:15.341509 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:15.341698 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:15.341570 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:15.560764 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:15.560736 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdsct" event={"ID":"fc030ff5-217f-41e5-8b48-bc45036de749","Type":"ContainerStarted","Data":"b3fa9edb20d6289b7633496bee5f1f327ca1bf4f69f674c712b93c38a3d67f53"} Sep 09 17:19:16.342445 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:16.342418 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:16.342677 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:16.342506 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:16.564937 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:16.564904 2553 generic.go:356] "Generic (PLEG): container finished" podID="fc030ff5-217f-41e5-8b48-bc45036de749" containerID="b3fa9edb20d6289b7633496bee5f1f327ca1bf4f69f674c712b93c38a3d67f53" exitCode=0 Sep 09 17:19:16.565387 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:16.564989 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdsct" event={"ID":"fc030ff5-217f-41e5-8b48-bc45036de749","Type":"ContainerDied","Data":"b3fa9edb20d6289b7633496bee5f1f327ca1bf4f69f674c712b93c38a3d67f53"} Sep 09 17:19:17.341952 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:17.341924 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:17.342130 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:17.342036 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:17.342130 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:17.341925 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:17.342239 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:17.342160 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:18.342308 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:18.342277 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:18.342759 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:18.342403 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:19.342058 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:19.342024 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:19.342334 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:19.342036 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:19.342334 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:19.342147 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:19.342334 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:19.342233 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:19.973354 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:19.973317 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:19.973551 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:19.973379 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:19.973551 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:19.973488 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:19:19.973694 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:19.973580 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret podName:22945efa-5c74-4bb9-b708-41a486a7e7c8 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:51.973559469 +0000 UTC m=+66.213465821 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret") pod "global-pull-secret-syncer-zlhb4" (UID: "22945efa-5c74-4bb9-b708-41a486a7e7c8") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:19:19.973694 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:19.973497 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:19:19.973694 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:19.973655 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs podName:8a9f88fb-c187-4672-bd8e-1916d5f9ff54 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:51.973639076 +0000 UTC m=+66.213545407 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs") pod "network-metrics-daemon-tc2sl" (UID: "8a9f88fb-c187-4672-bd8e-1916d5f9ff54") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:19:20.074700 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:20.074663 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dqf\" (UniqueName: \"kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf\") pod \"network-check-target-f6l6s\" (UID: \"7dc76395-0bd2-439e-b9f9-2c89105cc455\") " pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:20.074879 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:20.074817 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:19:20.074879 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:20.074841 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:19:20.074879 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:20.074856 2553 projected.go:196] Error preparing data for projected volume kube-api-access-k8dqf for pod openshift-network-diagnostics/network-check-target-f6l6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:19:20.075060 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:20.074906 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf podName:7dc76395-0bd2-439e-b9f9-2c89105cc455 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:52.074893617 +0000 UTC m=+66.314799961 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8dqf" (UniqueName: "kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf") pod "network-check-target-f6l6s" (UID: "7dc76395-0bd2-439e-b9f9-2c89105cc455") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:19:20.342462 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:20.342388 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:20.342876 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:20.342498 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:21.341958 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:21.341927 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:21.341958 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:21.341961 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:21.342139 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:21.342040 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:21.342178 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:21.342154 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:21.579004 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:21.578977 2553 generic.go:356] "Generic (PLEG): container finished" podID="fc030ff5-217f-41e5-8b48-bc45036de749" containerID="f40b7ba2ab7a6e70a012da6d92e967089befc86d9c3fee0cb998d33e7fe2c96b" exitCode=0 Sep 09 17:19:21.579389 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:21.579034 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdsct" event={"ID":"fc030ff5-217f-41e5-8b48-bc45036de749","Type":"ContainerDied","Data":"f40b7ba2ab7a6e70a012da6d92e967089befc86d9c3fee0cb998d33e7fe2c96b"} Sep 09 17:19:22.341926 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:22.341897 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:22.342082 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:22.342028 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:22.583953 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:22.583922 2553 generic.go:356] "Generic (PLEG): container finished" podID="fc030ff5-217f-41e5-8b48-bc45036de749" containerID="e7d46e6350b3b9c0ba346bf993521708041fc98fc8741be03a092fbca42f7a5a" exitCode=0 Sep 09 17:19:22.584304 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:22.583978 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdsct" event={"ID":"fc030ff5-217f-41e5-8b48-bc45036de749","Type":"ContainerDied","Data":"e7d46e6350b3b9c0ba346bf993521708041fc98fc8741be03a092fbca42f7a5a"} Sep 09 17:19:23.342439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:23.342267 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:23.342607 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:23.342332 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:23.342607 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:23.342501 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:23.342607 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:23.342575 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:23.589194 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:23.589162 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdsct" event={"ID":"fc030ff5-217f-41e5-8b48-bc45036de749","Type":"ContainerStarted","Data":"841a4b6d48defd742cf1f87dfddc3a1b3df5612cac612b2526860c02a5e6368d"} Sep 09 17:19:23.613584 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:23.613516 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wdsct" podStartSLOduration=8.163984236 podStartE2EDuration="37.613502998s" totalImagesPullingTime="29.449518762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:23.611836477 +0000 UTC m=+37.851742830" watchObservedRunningTime="2026-09-09 17:19:23.613502998 +0000 UTC m=+37.853409351" Sep 09 17:19:24.342035 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:24.342008 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:24.342189 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:24.342107 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:25.342172 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:25.342132 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:25.342574 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:25.342284 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:25.342574 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:25.342329 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:25.342574 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:25.342436 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:26.152101 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:26.152045 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zlhb4"] Sep 09 17:19:26.152292 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:26.152152 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:26.152292 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:26.152258 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:26.158256 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:26.158222 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f6l6s"] Sep 09 17:19:26.158387 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:26.158323 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc2sl"] Sep 09 17:19:26.158443 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:26.158436 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:26.158656 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:26.158614 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:26.158965 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:26.158945 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:26.159080 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:26.159061 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:28.342169 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:28.342136 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:28.342169 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:28.342155 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:28.342774 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:28.342139 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:28.342774 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:28.342238 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f6l6s" podUID="7dc76395-0bd2-439e-b9f9-2c89105cc455" Sep 09 17:19:28.342774 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:28.342302 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlhb4" podUID="22945efa-5c74-4bb9-b708-41a486a7e7c8" Sep 09 17:19:28.342774 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:28.342361 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc2sl" podUID="8a9f88fb-c187-4672-bd8e-1916d5f9ff54" Sep 09 17:19:30.117884 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.117854 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-213.ec2.internal" event="NodeReady" Sep 09 17:19:30.118249 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.117973 2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 17:19:30.158676 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.156834 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7894976f45-b8sps"] Sep 09 17:19:30.173581 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.173549 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7894976f45-b8sps"] Sep 09 17:19:30.173581 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.173584 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n6pqz"] Sep 09 17:19:30.173784 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.173687 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.176262 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.176237 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 17:19:30.176262 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.176249 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 17:19:30.176595 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.176580 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f2q7v\"" Sep 09 17:19:30.176768 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.176754 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 17:19:30.184019 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.184002 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 17:19:30.189690 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.189666 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n7mpt"] Sep 09 17:19:30.189801 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.189780 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.192514 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.192494 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 17:19:30.192615 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.192500 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 17:19:30.192615 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.192499 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pp6f6\"" Sep 09 17:19:30.212487 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.212468 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n7mpt"] Sep 09 17:19:30.212562 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.212496 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n6pqz"] Sep 09 17:19:30.212605 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.212577 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n7mpt" Sep 09 17:19:30.215123 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.215104 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m8lps\"" Sep 09 17:19:30.215206 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.215152 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 17:19:30.215256 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.215205 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 17:19:30.215295 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.215258 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 17:19:30.341598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.341544 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:30.341598 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.341559 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:30.341810 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.341544 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:30.344289 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.344272 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 17:19:30.344462 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.344378 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 17:19:30.344604 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.344587 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b9cf6\"" Sep 09 17:19:30.344686 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.344652 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 17:19:30.344790 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.344704 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 17:19:30.344790 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.344759 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-69pcj\"" Sep 09 17:19:30.353265 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353243 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e88c93fd-9a23-4b50-96f9-288af4d94959-ca-trust-extracted\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.353355 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353278 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e88c93fd-9a23-4b50-96f9-288af4d94959-bound-sa-token\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.353355 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353316 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e88c93fd-9a23-4b50-96f9-288af4d94959-registry-tls\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.353355 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353339 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e88c93fd-9a23-4b50-96f9-288af4d94959-registry-certificates\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.353473 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353362 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef63ca1-b605-4e44-a748-1319401e5e75-tmp-dir\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.353473 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353411 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e88c93fd-9a23-4b50-96f9-288af4d94959-trusted-ca\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.353473 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353430 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef63ca1-b605-4e44-a748-1319401e5e75-config-volume\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.353473 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353451 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/292acb04-43b0-4a43-a8c4-5ae8da4cae31-cert\") pod \"ingress-canary-n7mpt\" (UID: \"292acb04-43b0-4a43-a8c4-5ae8da4cae31\") " pod="openshift-ingress-canary/ingress-canary-n7mpt" Sep 09 17:19:30.353643 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353473 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e88c93fd-9a23-4b50-96f9-288af4d94959-image-registry-private-configuration\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.353643 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353495 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ef63ca1-b605-4e44-a748-1319401e5e75-metrics-tls\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.353643 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353526 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm9mw\" (UniqueName: \"kubernetes.io/projected/2ef63ca1-b605-4e44-a748-1319401e5e75-kube-api-access-sm9mw\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.353643 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353562 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e88c93fd-9a23-4b50-96f9-288af4d94959-installation-pull-secrets\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.353643 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353590 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8wn\" (UniqueName: \"kubernetes.io/projected/292acb04-43b0-4a43-a8c4-5ae8da4cae31-kube-api-access-dc8wn\") pod \"ingress-canary-n7mpt\" (UID: \"292acb04-43b0-4a43-a8c4-5ae8da4cae31\") " pod="openshift-ingress-canary/ingress-canary-n7mpt" Sep 09 17:19:30.353643 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.353604 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p26vs\" (UniqueName: \"kubernetes.io/projected/e88c93fd-9a23-4b50-96f9-288af4d94959-kube-api-access-p26vs\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.454881 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.454850 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8wn\" (UniqueName: \"kubernetes.io/projected/292acb04-43b0-4a43-a8c4-5ae8da4cae31-kube-api-access-dc8wn\") pod \"ingress-canary-n7mpt\" (UID: \"292acb04-43b0-4a43-a8c4-5ae8da4cae31\") " pod="openshift-ingress-canary/ingress-canary-n7mpt" Sep 09 17:19:30.454881 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.454880 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p26vs\" (UniqueName: \"kubernetes.io/projected/e88c93fd-9a23-4b50-96f9-288af4d94959-kube-api-access-p26vs\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.455100 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.454917 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e88c93fd-9a23-4b50-96f9-288af4d94959-ca-trust-extracted\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.455100 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.454937 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e88c93fd-9a23-4b50-96f9-288af4d94959-bound-sa-token\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.455100 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.454970 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e88c93fd-9a23-4b50-96f9-288af4d94959-registry-tls\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.455100 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455014 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e88c93fd-9a23-4b50-96f9-288af4d94959-registry-certificates\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.455301 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455159 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef63ca1-b605-4e44-a748-1319401e5e75-tmp-dir\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.455301 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455200 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e88c93fd-9a23-4b50-96f9-288af4d94959-trusted-ca\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.455301 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455225 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef63ca1-b605-4e44-a748-1319401e5e75-config-volume\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.455301 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455255 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/292acb04-43b0-4a43-a8c4-5ae8da4cae31-cert\") pod \"ingress-canary-n7mpt\" (UID: \"292acb04-43b0-4a43-a8c4-5ae8da4cae31\") " pod="openshift-ingress-canary/ingress-canary-n7mpt" Sep 09 17:19:30.455301 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455290 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e88c93fd-9a23-4b50-96f9-288af4d94959-image-registry-private-configuration\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.455542 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455319 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ef63ca1-b605-4e44-a748-1319401e5e75-metrics-tls\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.455542 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455323 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e88c93fd-9a23-4b50-96f9-288af4d94959-ca-trust-extracted\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.455542 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455360 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9mw\" (UniqueName: \"kubernetes.io/projected/2ef63ca1-b605-4e44-a748-1319401e5e75-kube-api-access-sm9mw\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.455542 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455393 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e88c93fd-9a23-4b50-96f9-288af4d94959-installation-pull-secrets\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.455542 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455528 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef63ca1-b605-4e44-a748-1319401e5e75-tmp-dir\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.456113 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.455899 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e88c93fd-9a23-4b50-96f9-288af4d94959-registry-certificates\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.456113 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.456062 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e88c93fd-9a23-4b50-96f9-288af4d94959-trusted-ca\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.456261 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.456117 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef63ca1-b605-4e44-a748-1319401e5e75-config-volume\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.459153 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.459131 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e88c93fd-9a23-4b50-96f9-288af4d94959-registry-tls\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.459252 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.459239 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e88c93fd-9a23-4b50-96f9-288af4d94959-image-registry-private-configuration\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.459322 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.459248 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ef63ca1-b605-4e44-a748-1319401e5e75-metrics-tls\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.459378 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.459319 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e88c93fd-9a23-4b50-96f9-288af4d94959-installation-pull-secrets\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.459378 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.459359 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/292acb04-43b0-4a43-a8c4-5ae8da4cae31-cert\") pod \"ingress-canary-n7mpt\" (UID: \"292acb04-43b0-4a43-a8c4-5ae8da4cae31\") " pod="openshift-ingress-canary/ingress-canary-n7mpt" Sep 09 17:19:30.463830 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.463806 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9mw\" (UniqueName: \"kubernetes.io/projected/2ef63ca1-b605-4e44-a748-1319401e5e75-kube-api-access-sm9mw\") pod \"dns-default-n6pqz\" (UID: \"2ef63ca1-b605-4e44-a748-1319401e5e75\") " pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.463932 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.463873 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p26vs\" (UniqueName: \"kubernetes.io/projected/e88c93fd-9a23-4b50-96f9-288af4d94959-kube-api-access-p26vs\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.464499 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.464484 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e88c93fd-9a23-4b50-96f9-288af4d94959-bound-sa-token\") pod \"image-registry-7894976f45-b8sps\" (UID: \"e88c93fd-9a23-4b50-96f9-288af4d94959\") " pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.464499 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.464493 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8wn\" (UniqueName: \"kubernetes.io/projected/292acb04-43b0-4a43-a8c4-5ae8da4cae31-kube-api-access-dc8wn\") pod \"ingress-canary-n7mpt\" (UID: \"292acb04-43b0-4a43-a8c4-5ae8da4cae31\") " pod="openshift-ingress-canary/ingress-canary-n7mpt" Sep 09 17:19:30.487568 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.487552 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:30.497628 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.497597 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:30.520481 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.520460 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n7mpt" Sep 09 17:19:30.819448 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.819412 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n6pqz"] Sep 09 17:19:30.820252 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.820228 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7894976f45-b8sps"] Sep 09 17:19:30.820986 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:19:30.820961 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef63ca1_b605_4e44_a748_1319401e5e75.slice/crio-3d8870e0ce8197c7141b335de6416ab66ccb124dec0356d245eb0f4ddc88a3da WatchSource:0}: Error finding container 3d8870e0ce8197c7141b335de6416ab66ccb124dec0356d245eb0f4ddc88a3da: Status 404 returned error can't find the container with id 3d8870e0ce8197c7141b335de6416ab66ccb124dec0356d245eb0f4ddc88a3da Sep 09 17:19:30.822027 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:19:30.822003 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88c93fd_9a23_4b50_96f9_288af4d94959.slice/crio-a082a13f2be6f9d6c138eab3f122259b49e56f525038b30c5371ef468ce1b326 WatchSource:0}: Error finding container a082a13f2be6f9d6c138eab3f122259b49e56f525038b30c5371ef468ce1b326: Status 404 returned error can't find the container with id a082a13f2be6f9d6c138eab3f122259b49e56f525038b30c5371ef468ce1b326 Sep 09 17:19:30.832785 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:30.832765 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n7mpt"] Sep 09 17:19:30.835928 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:19:30.835900 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292acb04_43b0_4a43_a8c4_5ae8da4cae31.slice/crio-4e5988e07a52df0042b419f3936aec91d6fdfd4d19bc25e45ddc99c0b0be393b WatchSource:0}: Error finding container 4e5988e07a52df0042b419f3936aec91d6fdfd4d19bc25e45ddc99c0b0be393b: Status 404 returned error can't find the container with id 4e5988e07a52df0042b419f3936aec91d6fdfd4d19bc25e45ddc99c0b0be393b Sep 09 17:19:31.611853 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:31.611646 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n7mpt" event={"ID":"292acb04-43b0-4a43-a8c4-5ae8da4cae31","Type":"ContainerStarted","Data":"4e5988e07a52df0042b419f3936aec91d6fdfd4d19bc25e45ddc99c0b0be393b"} Sep 09 17:19:31.612931 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:31.612903 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n6pqz" event={"ID":"2ef63ca1-b605-4e44-a748-1319401e5e75","Type":"ContainerStarted","Data":"3d8870e0ce8197c7141b335de6416ab66ccb124dec0356d245eb0f4ddc88a3da"} Sep 09 17:19:31.614497 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:31.614473 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7894976f45-b8sps" event={"ID":"e88c93fd-9a23-4b50-96f9-288af4d94959","Type":"ContainerStarted","Data":"a9156a45ba32fb13aeaff9fe3cf163da36897940b660aef37d9ea3ec0adf8dbd"} Sep 09 17:19:31.614580 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:31.614512 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7894976f45-b8sps" event={"ID":"e88c93fd-9a23-4b50-96f9-288af4d94959","Type":"ContainerStarted","Data":"a082a13f2be6f9d6c138eab3f122259b49e56f525038b30c5371ef468ce1b326"} Sep 09 17:19:31.614787 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:31.614763 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:33.621955 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:33.621918 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n6pqz" event={"ID":"2ef63ca1-b605-4e44-a748-1319401e5e75","Type":"ContainerStarted","Data":"bb1b982f18588c4c15e14f52a8df1c83406867040cdcbad2a6b25fadefd7f0f1"} Sep 09 17:19:33.622435 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:33.621983 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n6pqz" event={"ID":"2ef63ca1-b605-4e44-a748-1319401e5e75","Type":"ContainerStarted","Data":"38e2792d4d9c3286d9fcaf1907cc3d4a65ae4bb30148d3b3e2586a514c9b0028"} Sep 09 17:19:33.622435 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:33.622010 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:33.644010 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:33.643963 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7894976f45-b8sps" podStartSLOduration=8.643947302 podStartE2EDuration="8.643947302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:31.635709993 +0000 UTC m=+45.875616346" watchObservedRunningTime="2026-09-09 17:19:33.643947302 +0000 UTC m=+47.883853654" Sep 09 17:19:33.644310 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:33.644281 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n6pqz" podStartSLOduration=1.650407746 podStartE2EDuration="3.644270014s" totalImagesPullingTime="1.993862268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:33.64278812 +0000 UTC m=+47.882694484" watchObservedRunningTime="2026-09-09 17:19:33.644270014 +0000 UTC m=+47.884176401" Sep 09 17:19:34.626572 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:34.626482 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n7mpt" event={"ID":"292acb04-43b0-4a43-a8c4-5ae8da4cae31","Type":"ContainerStarted","Data":"e2cf79b8df9c4181b4fa056efde0e47dcfd8061ee24f0c60a7c7a4138c3388fb"} Sep 09 17:19:34.641310 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:34.641263 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n7mpt" podStartSLOduration=1.251338138 podStartE2EDuration="4.641249195s" totalImagesPullingTime="3.389911057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:34.640708584 +0000 UTC m=+48.880614939" watchObservedRunningTime="2026-09-09 17:19:34.641249195 +0000 UTC m=+48.881155547" Sep 09 17:19:37.318300 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.318254 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2vdjr"] Sep 09 17:19:37.330396 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.330375 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.334480 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.334457 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2vdjr"] Sep 09 17:19:37.335030 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.334868 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 17:19:37.335030 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.334905 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 17:19:37.335030 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.334916 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 17:19:37.335030 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.334964 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 17:19:37.335030 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.334985 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ph5lx\"" Sep 09 17:19:37.408063 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.408036 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fdgc\" (UniqueName: \"kubernetes.io/projected/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-api-access-2fdgc\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.408206 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.408093 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-crio-socket\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.408206 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.408110 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.408206 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.408151 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.408206 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.408169 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-data-volume\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.508458 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.508420 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-crio-socket\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.508617 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.508463 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.508617 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.508499 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.508617 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.508523 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-data-volume\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.508617 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.508567 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fdgc\" (UniqueName: \"kubernetes.io/projected/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-api-access-2fdgc\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.508617 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:37.508592 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 17:19:37.508909 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:37.508651 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 17:19:37.508909 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.508672 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-crio-socket\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.508909 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:37.508687 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls podName:4c3f11a8-d34e-4c31-818d-5bd2d17fe053 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:38.008664751 +0000 UTC m=+52.248571087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vdjr" (UID: "4c3f11a8-d34e-4c31-818d-5bd2d17fe053") : secret "insights-runtime-extractor-tls" not found Sep 09 17:19:37.508909 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:37.508733 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm podName:4c3f11a8-d34e-4c31-818d-5bd2d17fe053 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:38.008714276 +0000 UTC m=+52.248620639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vdjr" (UID: "4c3f11a8-d34e-4c31-818d-5bd2d17fe053") : configmap "kube-rbac-proxy" not found Sep 09 17:19:37.509154 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.509132 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-data-volume\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:37.520575 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:37.520550 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fdgc\" (UniqueName: \"kubernetes.io/projected/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-api-access-2fdgc\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:38.012064 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:38.012031 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:38.012064 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:38.012072 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:38.012261 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:38.012164 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 17:19:38.012261 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:38.012166 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 17:19:38.012261 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:38.012227 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm podName:4c3f11a8-d34e-4c31-818d-5bd2d17fe053 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:39.012212357 +0000 UTC m=+53.252118688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vdjr" (UID: "4c3f11a8-d34e-4c31-818d-5bd2d17fe053") : configmap "kube-rbac-proxy" not found Sep 09 17:19:38.012261 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:38.012240 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls podName:4c3f11a8-d34e-4c31-818d-5bd2d17fe053 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:39.012234229 +0000 UTC m=+53.252140560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vdjr" (UID: "4c3f11a8-d34e-4c31-818d-5bd2d17fe053") : secret "insights-runtime-extractor-tls" not found Sep 09 17:19:38.688513 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:38.688486 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n6pqz_2ef63ca1-b605-4e44-a748-1319401e5e75/dns/0.log" Sep 09 17:19:38.868064 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:38.868043 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n6pqz_2ef63ca1-b605-4e44-a748-1319401e5e75/kube-rbac-proxy/0.log" Sep 09 17:19:39.017243 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:39.017167 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:39.017243 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:39.017211 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:39.017414 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:39.017296 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 17:19:39.017414 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:39.017307 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 17:19:39.017414 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:39.017356 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm podName:4c3f11a8-d34e-4c31-818d-5bd2d17fe053 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:41.017337645 +0000 UTC m=+55.257243976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vdjr" (UID: "4c3f11a8-d34e-4c31-818d-5bd2d17fe053") : configmap "kube-rbac-proxy" not found Sep 09 17:19:39.017414 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:39.017369 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls podName:4c3f11a8-d34e-4c31-818d-5bd2d17fe053 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:41.017363574 +0000 UTC m=+55.257269905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vdjr" (UID: "4c3f11a8-d34e-4c31-818d-5bd2d17fe053") : secret "insights-runtime-extractor-tls" not found Sep 09 17:19:39.071595 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:39.071568 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-54x4s_b88c2732-169f-47a7-b117-b556624997e7/dns-node-resolver/0.log" Sep 09 17:19:40.072072 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:40.072042 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7894976f45-b8sps_e88c93fd-9a23-4b50-96f9-288af4d94959/registry/0.log" Sep 09 17:19:40.668082 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:40.668053 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tt8b9_2a5ff466-6997-438f-8364-7277a690051d/node-ca/0.log" Sep 09 17:19:41.031877 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:41.031792 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:41.031877 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:41.031836 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:41.032062 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:41.031917 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 17:19:41.032062 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:41.031954 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 17:19:41.032062 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:41.032012 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls podName:4c3f11a8-d34e-4c31-818d-5bd2d17fe053 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:45.031994416 +0000 UTC m=+59.271900783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vdjr" (UID: "4c3f11a8-d34e-4c31-818d-5bd2d17fe053") : secret "insights-runtime-extractor-tls" not found Sep 09 17:19:41.032062 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:41.032027 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm podName:4c3f11a8-d34e-4c31-818d-5bd2d17fe053 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:45.03202065 +0000 UTC m=+59.271926984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vdjr" (UID: "4c3f11a8-d34e-4c31-818d-5bd2d17fe053") : configmap "kube-rbac-proxy" not found Sep 09 17:19:41.468936 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:41.468872 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n7mpt_292acb04-43b0-4a43-a8c4-5ae8da4cae31/serve-healthcheck-canary/0.log" Sep 09 17:19:43.628715 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:43.628681 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n6pqz" Sep 09 17:19:45.058959 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:45.058917 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:45.058959 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:45.058963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:45.059362 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:45.059030 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 17:19:45.059362 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:45.059061 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 17:19:45.059362 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:45.059084 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm podName:4c3f11a8-d34e-4c31-818d-5bd2d17fe053 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:53.059070948 +0000 UTC m=+67.298977280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vdjr" (UID: "4c3f11a8-d34e-4c31-818d-5bd2d17fe053") : configmap "kube-rbac-proxy" not found Sep 09 17:19:45.059362 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:45.059115 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls podName:4c3f11a8-d34e-4c31-818d-5bd2d17fe053 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:53.059101321 +0000 UTC m=+67.299007651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vdjr" (UID: "4c3f11a8-d34e-4c31-818d-5bd2d17fe053") : secret "insights-runtime-extractor-tls" not found Sep 09 17:19:46.575368 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.575340 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkdmr" Sep 09 17:19:46.700408 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.700378 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x2lqb"] Sep 09 17:19:46.703140 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.703123 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.706224 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.706203 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 17:19:46.706490 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.706467 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 17:19:46.706556 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.706531 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 17:19:46.706556 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.706473 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 17:19:46.706556 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.706545 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 17:19:46.706707 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.706646 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gn8g8\"" Sep 09 17:19:46.706885 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.706872 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 17:19:46.771262 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.771239 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-tls\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.771369 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.771271 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c03dc487-852a-4d46-b563-1e2e9c3c68b3-root\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.771369 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.771285 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-wtmp\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.771369 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.771302 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c03dc487-852a-4d46-b563-1e2e9c3c68b3-metrics-client-ca\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.771491 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.771373 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-textfile\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.771491 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.771404 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.771491 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.771422 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5r5l\" (UniqueName: \"kubernetes.io/projected/c03dc487-852a-4d46-b563-1e2e9c3c68b3-kube-api-access-l5r5l\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.771491 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.771449 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c03dc487-852a-4d46-b563-1e2e9c3c68b3-sys\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.771491 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.771464 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.871970 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.871913 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-tls\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.871970 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.871943 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c03dc487-852a-4d46-b563-1e2e9c3c68b3-root\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.871970 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.871958 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-wtmp\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872140 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.871973 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c03dc487-852a-4d46-b563-1e2e9c3c68b3-metrics-client-ca\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872140 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.872007 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-textfile\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872140 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.872014 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c03dc487-852a-4d46-b563-1e2e9c3c68b3-root\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872140 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.872030 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872140 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.872051 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5r5l\" (UniqueName: \"kubernetes.io/projected/c03dc487-852a-4d46-b563-1e2e9c3c68b3-kube-api-access-l5r5l\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872140 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:46.872085 2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 17:19:46.872140 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.872088 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c03dc487-852a-4d46-b563-1e2e9c3c68b3-sys\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872140 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.872121 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-wtmp\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872140 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.872133 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872541 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:46.872149 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-tls podName:c03dc487-852a-4d46-b563-1e2e9c3c68b3 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:47.372129652 +0000 UTC m=+61.612036007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-tls") pod "node-exporter-x2lqb" (UID: "c03dc487-852a-4d46-b563-1e2e9c3c68b3") : secret "node-exporter-tls" not found Sep 09 17:19:46.872541 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.872176 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c03dc487-852a-4d46-b563-1e2e9c3c68b3-sys\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872541 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.872351 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-textfile\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872694 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.872669 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.872728 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.872690 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c03dc487-852a-4d46-b563-1e2e9c3c68b3-metrics-client-ca\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.874594 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.874573 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:46.881262 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:46.881238 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5r5l\" (UniqueName: \"kubernetes.io/projected/c03dc487-852a-4d46-b563-1e2e9c3c68b3-kube-api-access-l5r5l\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:47.375897 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.375850 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-tls\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:47.378156 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.378139 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c03dc487-852a-4d46-b563-1e2e9c3c68b3-node-exporter-tls\") pod \"node-exporter-x2lqb\" (UID: \"c03dc487-852a-4d46-b563-1e2e9c3c68b3\") " pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:47.605554 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.605520 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 17:19:47.608383 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.608366 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.611478 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.611459 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x2lqb" Sep 09 17:19:47.615769 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.615416 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 17:19:47.615769 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.615568 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 17:19:47.615769 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.615640 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 17:19:47.615943 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.615909 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 17:19:47.620726 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.620710 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 17:19:47.621743 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:19:47.621723 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03dc487_852a_4d46_b563_1e2e9c3c68b3.slice/crio-e7c2b1e4bc6b8c4652fd10ae6ac4d0072e4044cd8ed9a1fa508e1d0c58ebef62 WatchSource:0}: Error finding container e7c2b1e4bc6b8c4652fd10ae6ac4d0072e4044cd8ed9a1fa508e1d0c58ebef62: Status 404 returned error can't find the container with id e7c2b1e4bc6b8c4652fd10ae6ac4d0072e4044cd8ed9a1fa508e1d0c58ebef62 Sep 09 17:19:47.621856 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.621753 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 17:19:47.622199 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.622183 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 17:19:47.624824 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.622385 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 17:19:47.624824 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.622421 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 17:19:47.624824 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.622998 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sjmmb\"" Sep 09 17:19:47.637361 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.637339 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 17:19:47.657684 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.657655 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x2lqb" event={"ID":"c03dc487-852a-4d46-b563-1e2e9c3c68b3","Type":"ContainerStarted","Data":"e7c2b1e4bc6b8c4652fd10ae6ac4d0072e4044cd8ed9a1fa508e1d0c58ebef62"} Sep 09 17:19:47.678048 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678026 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-web-config\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678141 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678066 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678141 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678094 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678206 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678166 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678206 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678197 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678272 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678214 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-config-out\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678272 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678234 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678272 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678260 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678356 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678295 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678356 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678328 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678361 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678419 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.678523 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.678446 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22sgj\" (UniqueName: \"kubernetes.io/projected/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-kube-api-access-22sgj\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.779533 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.779508 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.779724 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.779549 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.779724 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.779592 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.779724 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.779666 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22sgj\" (UniqueName: \"kubernetes.io/projected/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-kube-api-access-22sgj\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.779724 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.779715 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-web-config\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.779944 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.779755 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.779944 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.779927 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.780048 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.780021 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.780102 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.780053 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.780102 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.780078 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-config-out\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.780203 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.780101 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.780203 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.780128 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.780203 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.780180 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.781044 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.780874 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.781044 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.780983 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.781044 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:47.781001 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-alertmanager-trusted-ca-bundle podName:9e6b1781-a2cb-491a-a6e0-37fbe045bbde nodeName:}" failed. No retries permitted until 2026-09-09 17:19:48.280980691 +0000 UTC m=+62.520887023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "9e6b1781-a2cb-491a-a6e0-37fbe045bbde") : configmap references non-existent config key: ca-bundle.crt Sep 09 17:19:47.782823 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.782779 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.782929 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.782853 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.783169 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.783151 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-config-out\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.783234 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.783214 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-web-config\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.783357 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.783339 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.783496 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.783478 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.783578 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.783555 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.783679 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.783662 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.784329 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.784307 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:47.787891 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:47.787870 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22sgj\" (UniqueName: \"kubernetes.io/projected/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-kube-api-access-22sgj\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:48.283367 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:48.283330 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:48.284231 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:48.284206 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e6b1781-a2cb-491a-a6e0-37fbe045bbde-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e6b1781-a2cb-491a-a6e0-37fbe045bbde\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:48.517897 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:48.517863 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:19:48.986811 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:48.986785 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 17:19:48.988307 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:19:48.988269 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6b1781_a2cb_491a_a6e0_37fbe045bbde.slice/crio-9e9df0e296fb7451a4d03cad154452e7fbb750b4ed7c537ae6a527b00552ac6c WatchSource:0}: Error finding container 9e9df0e296fb7451a4d03cad154452e7fbb750b4ed7c537ae6a527b00552ac6c: Status 404 returned error can't find the container with id 9e9df0e296fb7451a4d03cad154452e7fbb750b4ed7c537ae6a527b00552ac6c Sep 09 17:19:49.577754 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.577726 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp"] Sep 09 17:19:49.580138 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.580119 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.583316 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.583291 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 09 17:19:49.583409 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.583382 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 09 17:19:49.583847 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.583826 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 09 17:19:49.584180 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.584164 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9slon6k0og698\"" Sep 09 17:19:49.584224 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.584209 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 09 17:19:49.584277 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.584212 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 09 17:19:49.584347 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.584262 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bdjj6\"" Sep 09 17:19:49.594704 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.594676 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.594833 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.594813 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sx7z\" (UniqueName: \"kubernetes.io/projected/abc4748f-3749-4a07-941f-9ba773ab5e8f-kube-api-access-8sx7z\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.594897 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.594852 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.594897 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.594821 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp"] Sep 09 17:19:49.594990 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.594922 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-tls\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.594990 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.594958 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.595093 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.595008 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abc4748f-3749-4a07-941f-9ba773ab5e8f-metrics-client-ca\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.595093 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.595059 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-grpc-tls\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.595180 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.595099 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.665182 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.665153 2553 generic.go:356] "Generic (PLEG): container finished" podID="c03dc487-852a-4d46-b563-1e2e9c3c68b3" containerID="40b14aec907abea3a004ca94dfd7fbb6f9bb71ed4ccfa9a2b3e07603b6ef84cf" exitCode=0 Sep 09 17:19:49.665312 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.665243 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x2lqb" event={"ID":"c03dc487-852a-4d46-b563-1e2e9c3c68b3","Type":"ContainerDied","Data":"40b14aec907abea3a004ca94dfd7fbb6f9bb71ed4ccfa9a2b3e07603b6ef84cf"} Sep 09 17:19:49.666462 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.666438 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e6b1781-a2cb-491a-a6e0-37fbe045bbde","Type":"ContainerStarted","Data":"9e9df0e296fb7451a4d03cad154452e7fbb750b4ed7c537ae6a527b00552ac6c"} Sep 09 17:19:49.695779 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.695755 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sx7z\" (UniqueName: \"kubernetes.io/projected/abc4748f-3749-4a07-941f-9ba773ab5e8f-kube-api-access-8sx7z\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.695878 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.695785 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.695878 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.695821 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-tls\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.695878 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.695857 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.696040 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.695900 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abc4748f-3749-4a07-941f-9ba773ab5e8f-metrics-client-ca\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.696040 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.695956 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-grpc-tls\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.696040 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.695991 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.696188 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.696044 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.696614 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.696588 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abc4748f-3749-4a07-941f-9ba773ab5e8f-metrics-client-ca\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.698173 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.698154 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.698299 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.698276 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.698671 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.698643 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-tls\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.698848 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.698831 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-grpc-tls\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.698916 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.698856 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.698916 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.698878 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/abc4748f-3749-4a07-941f-9ba773ab5e8f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.704079 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.704057 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sx7z\" (UniqueName: \"kubernetes.io/projected/abc4748f-3749-4a07-941f-9ba773ab5e8f-kube-api-access-8sx7z\") pod \"thanos-querier-8ddf9ddbd-jzljp\" (UID: \"abc4748f-3749-4a07-941f-9ba773ab5e8f\") " pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:49.889293 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:49.889223 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:50.171334 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:50.171303 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp"] Sep 09 17:19:50.339828 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:19:50.339790 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc4748f_3749_4a07_941f_9ba773ab5e8f.slice/crio-9be1f7c06827786fda5e7b82636079044c0df70313d40dcf425277a8621c4c26 WatchSource:0}: Error finding container 9be1f7c06827786fda5e7b82636079044c0df70313d40dcf425277a8621c4c26: Status 404 returned error can't find the container with id 9be1f7c06827786fda5e7b82636079044c0df70313d40dcf425277a8621c4c26 Sep 09 17:19:50.670306 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:50.670276 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" event={"ID":"abc4748f-3749-4a07-941f-9ba773ab5e8f","Type":"ContainerStarted","Data":"9be1f7c06827786fda5e7b82636079044c0df70313d40dcf425277a8621c4c26"} Sep 09 17:19:50.672521 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:50.672498 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x2lqb" event={"ID":"c03dc487-852a-4d46-b563-1e2e9c3c68b3","Type":"ContainerStarted","Data":"bbbde419f4ace2e8a7e74f163d3032d8100a4f33c600cf9eb524b14fb4dd5551"} Sep 09 17:19:50.672661 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:50.672544 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x2lqb" event={"ID":"c03dc487-852a-4d46-b563-1e2e9c3c68b3","Type":"ContainerStarted","Data":"3d57108b889cf1b37f4775e707e2e5ace49c21398fc1bf3396ff3c529028b688"} Sep 09 17:19:50.693018 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:50.692965 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x2lqb" podStartSLOduration=3.524271292 podStartE2EDuration="4.69294662s" totalImagesPullingTime="1.106231505s" totalInitContainerRuntime="62.443823ms" isStatefulPod=true podCreationTimestamp="2026-09-09 17:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:50.692098719 +0000 UTC m=+64.932005072" watchObservedRunningTime="2026-09-09 17:19:50.69294662 +0000 UTC m=+64.932852974" Sep 09 17:19:51.011551 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.011517 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84bbb9896d-75tsk"] Sep 09 17:19:51.025146 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.025118 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.028041 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.028018 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7ecc1ds5lcr88\"" Sep 09 17:19:51.028161 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.028060 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84bbb9896d-75tsk"] Sep 09 17:19:51.029129 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.029112 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 09 17:19:51.029699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.029681 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 09 17:19:51.029797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.029775 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 09 17:19:51.029797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.029788 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s5bth\"" Sep 09 17:19:51.029901 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.029796 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 17:19:51.107827 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.107793 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/efc30c56-e147-45f4-b69c-5d712eb218b1-secret-metrics-server-tls\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.107997 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.107846 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/efc30c56-e147-45f4-b69c-5d712eb218b1-secret-metrics-server-client-certs\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.107997 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.107919 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/efc30c56-e147-45f4-b69c-5d712eb218b1-audit-log\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.107997 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.107970 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/efc30c56-e147-45f4-b69c-5d712eb218b1-client-ca-bundle\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.107997 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.107996 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/efc30c56-e147-45f4-b69c-5d712eb218b1-metrics-server-audit-profiles\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.108213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.108062 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc30c56-e147-45f4-b69c-5d712eb218b1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.108213 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.108109 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9nd\" (UniqueName: \"kubernetes.io/projected/efc30c56-e147-45f4-b69c-5d712eb218b1-kube-api-access-2x9nd\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.209302 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.209271 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/efc30c56-e147-45f4-b69c-5d712eb218b1-secret-metrics-server-tls\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.209731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.209314 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/efc30c56-e147-45f4-b69c-5d712eb218b1-secret-metrics-server-client-certs\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.209731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.209344 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/efc30c56-e147-45f4-b69c-5d712eb218b1-audit-log\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.209731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.209403 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/efc30c56-e147-45f4-b69c-5d712eb218b1-client-ca-bundle\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.209731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.209441 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/efc30c56-e147-45f4-b69c-5d712eb218b1-metrics-server-audit-profiles\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.209731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.209471 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc30c56-e147-45f4-b69c-5d712eb218b1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.209731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.209499 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9nd\" (UniqueName: \"kubernetes.io/projected/efc30c56-e147-45f4-b69c-5d712eb218b1-kube-api-access-2x9nd\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.210004 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.209798 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/efc30c56-e147-45f4-b69c-5d712eb218b1-audit-log\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.210224 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.210200 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc30c56-e147-45f4-b69c-5d712eb218b1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.210661 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.210611 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/efc30c56-e147-45f4-b69c-5d712eb218b1-metrics-server-audit-profiles\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.212067 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.212043 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/efc30c56-e147-45f4-b69c-5d712eb218b1-secret-metrics-server-client-certs\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.212327 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.212308 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/efc30c56-e147-45f4-b69c-5d712eb218b1-secret-metrics-server-tls\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.212391 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.212315 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/efc30c56-e147-45f4-b69c-5d712eb218b1-client-ca-bundle\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.220830 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.220800 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9nd\" (UniqueName: \"kubernetes.io/projected/efc30c56-e147-45f4-b69c-5d712eb218b1-kube-api-access-2x9nd\") pod \"metrics-server-84bbb9896d-75tsk\" (UID: \"efc30c56-e147-45f4-b69c-5d712eb218b1\") " pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.265410 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.265355 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl"] Sep 09 17:19:51.278364 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.278343 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl"] Sep 09 17:19:51.278467 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.278422 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" Sep 09 17:19:51.281154 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.281132 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 09 17:19:51.281253 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.281190 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ntc4m\"" Sep 09 17:19:51.310294 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.310271 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqzx\" (UniqueName: \"kubernetes.io/projected/a664f0f0-5cc2-421f-bc9f-d35320a559ab-kube-api-access-nlqzx\") pod \"monitoring-plugin-85bbbbbf4-5x4fl\" (UID: \"a664f0f0-5cc2-421f-bc9f-d35320a559ab\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" Sep 09 17:19:51.310370 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.310299 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a664f0f0-5cc2-421f-bc9f-d35320a559ab-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-5x4fl\" (UID: \"a664f0f0-5cc2-421f-bc9f-d35320a559ab\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" Sep 09 17:19:51.334804 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.334784 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:19:51.411468 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.411438 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqzx\" (UniqueName: \"kubernetes.io/projected/a664f0f0-5cc2-421f-bc9f-d35320a559ab-kube-api-access-nlqzx\") pod \"monitoring-plugin-85bbbbbf4-5x4fl\" (UID: \"a664f0f0-5cc2-421f-bc9f-d35320a559ab\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" Sep 09 17:19:51.411606 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.411485 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a664f0f0-5cc2-421f-bc9f-d35320a559ab-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-5x4fl\" (UID: \"a664f0f0-5cc2-421f-bc9f-d35320a559ab\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" Sep 09 17:19:51.411708 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:51.411683 2553 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 09 17:19:51.411816 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:19:51.411774 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a664f0f0-5cc2-421f-bc9f-d35320a559ab-monitoring-plugin-cert podName:a664f0f0-5cc2-421f-bc9f-d35320a559ab nodeName:}" failed. No retries permitted until 2026-09-09 17:19:51.911753314 +0000 UTC m=+66.151659647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a664f0f0-5cc2-421f-bc9f-d35320a559ab-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-5x4fl" (UID: "a664f0f0-5cc2-421f-bc9f-d35320a559ab") : secret "monitoring-plugin-cert" not found Sep 09 17:19:51.420831 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.420808 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqzx\" (UniqueName: \"kubernetes.io/projected/a664f0f0-5cc2-421f-bc9f-d35320a559ab-kube-api-access-nlqzx\") pod \"monitoring-plugin-85bbbbbf4-5x4fl\" (UID: \"a664f0f0-5cc2-421f-bc9f-d35320a559ab\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" Sep 09 17:19:51.613514 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.613279 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84bbb9896d-75tsk"] Sep 09 17:19:51.617599 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:19:51.617568 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc30c56_e147_45f4_b69c_5d712eb218b1.slice/crio-9f0e25da4b26e4d5b122b18c7c70680a85a0b1260cadd1c6dee639ae8700fc9c WatchSource:0}: Error finding container 9f0e25da4b26e4d5b122b18c7c70680a85a0b1260cadd1c6dee639ae8700fc9c: Status 404 returned error can't find the container with id 9f0e25da4b26e4d5b122b18c7c70680a85a0b1260cadd1c6dee639ae8700fc9c Sep 09 17:19:51.678670 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.678638 2553 generic.go:356] "Generic (PLEG): container finished" podID="9e6b1781-a2cb-491a-a6e0-37fbe045bbde" containerID="db35ade8848c9af370c7c9d6a61fe0c616865532dcc9a8736f1614d83c871d3e" exitCode=0 Sep 09 17:19:51.678807 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.678764 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e6b1781-a2cb-491a-a6e0-37fbe045bbde","Type":"ContainerDied","Data":"db35ade8848c9af370c7c9d6a61fe0c616865532dcc9a8736f1614d83c871d3e"} Sep 09 17:19:51.680571 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.680443 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" event={"ID":"efc30c56-e147-45f4-b69c-5d712eb218b1","Type":"ContainerStarted","Data":"9f0e25da4b26e4d5b122b18c7c70680a85a0b1260cadd1c6dee639ae8700fc9c"} Sep 09 17:19:51.915535 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.915490 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7c4798658f-z8qcd"] Sep 09 17:19:51.916444 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.916408 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a664f0f0-5cc2-421f-bc9f-d35320a559ab-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-5x4fl\" (UID: \"a664f0f0-5cc2-421f-bc9f-d35320a559ab\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" Sep 09 17:19:51.919176 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.919151 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a664f0f0-5cc2-421f-bc9f-d35320a559ab-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-5x4fl\" (UID: \"a664f0f0-5cc2-421f-bc9f-d35320a559ab\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" Sep 09 17:19:51.937344 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.937322 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c4798658f-z8qcd"] Sep 09 17:19:51.937483 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.937451 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:51.941804 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.941782 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 09 17:19:51.942200 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.942177 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 09 17:19:51.942317 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.942228 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8dncd\"" Sep 09 17:19:51.942479 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.942461 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 09 17:19:51.942724 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.942658 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 09 17:19:51.942724 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.942678 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 09 17:19:51.949748 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:51.949728 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 09 17:19:52.017908 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.017796 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:52.017908 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.017855 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f75adb-8242-4ad5-af1b-41f6df199e27-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.018124 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.017974 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f75adb-8242-4ad5-af1b-41f6df199e27-metrics-client-ca\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.018124 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.018018 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-federate-client-tls\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.018124 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.018053 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.018252 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.018132 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-telemeter-client-tls\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.018252 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.018191 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f75adb-8242-4ad5-af1b-41f6df199e27-serving-certs-ca-bundle\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.018252 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.018216 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcntx\" (UniqueName: \"kubernetes.io/projected/e4f75adb-8242-4ad5-af1b-41f6df199e27-kube-api-access-wcntx\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.018252 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.018239 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-secret-telemeter-client\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.018500 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.018275 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:52.020551 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.020527 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 17:19:52.020886 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.020871 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 17:19:52.030536 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.030514 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9f88fb-c187-4672-bd8e-1916d5f9ff54-metrics-certs\") pod \"network-metrics-daemon-tc2sl\" (UID: \"8a9f88fb-c187-4672-bd8e-1916d5f9ff54\") " pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:52.030745 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.030727 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22945efa-5c74-4bb9-b708-41a486a7e7c8-original-pull-secret\") pod \"global-pull-secret-syncer-zlhb4\" (UID: \"22945efa-5c74-4bb9-b708-41a486a7e7c8\") " pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:52.119534 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.119505 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f75adb-8242-4ad5-af1b-41f6df199e27-metrics-client-ca\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.119689 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.119546 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-federate-client-tls\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.119689 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.119586 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.119689 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.119640 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-telemeter-client-tls\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.119862 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.119689 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f75adb-8242-4ad5-af1b-41f6df199e27-serving-certs-ca-bundle\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.119862 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.119721 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcntx\" (UniqueName: \"kubernetes.io/projected/e4f75adb-8242-4ad5-af1b-41f6df199e27-kube-api-access-wcntx\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.119862 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.119749 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-secret-telemeter-client\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.119862 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.119801 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dqf\" (UniqueName: \"kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf\") pod \"network-check-target-f6l6s\" (UID: \"7dc76395-0bd2-439e-b9f9-2c89105cc455\") " pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:52.120041 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.119861 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f75adb-8242-4ad5-af1b-41f6df199e27-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.120680 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.120331 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f75adb-8242-4ad5-af1b-41f6df199e27-metrics-client-ca\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.120680 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.120486 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f75adb-8242-4ad5-af1b-41f6df199e27-serving-certs-ca-bundle\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.120959 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.120933 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f75adb-8242-4ad5-af1b-41f6df199e27-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.122552 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.122530 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.122715 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.122697 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-telemeter-client-tls\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.123201 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.123183 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-federate-client-tls\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.123440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.123414 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e4f75adb-8242-4ad5-af1b-41f6df199e27-secret-telemeter-client\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.123720 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.123701 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 17:19:52.129300 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.129278 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcntx\" (UniqueName: \"kubernetes.io/projected/e4f75adb-8242-4ad5-af1b-41f6df199e27-kube-api-access-wcntx\") pod \"telemeter-client-7c4798658f-z8qcd\" (UID: \"e4f75adb-8242-4ad5-af1b-41f6df199e27\") " pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.133402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.133382 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 17:19:52.143976 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.143954 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8dqf\" (UniqueName: \"kubernetes.io/projected/7dc76395-0bd2-439e-b9f9-2c89105cc455-kube-api-access-k8dqf\") pod \"network-check-target-f6l6s\" (UID: \"7dc76395-0bd2-439e-b9f9-2c89105cc455\") " pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:52.186824 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.186775 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" Sep 09 17:19:52.250002 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.249977 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" Sep 09 17:19:52.251017 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.250993 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlhb4" Sep 09 17:19:52.258974 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.258952 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b9cf6\"" Sep 09 17:19:52.263255 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.263234 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-69pcj\"" Sep 09 17:19:52.267637 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.267596 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:52.271400 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.271377 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc2sl" Sep 09 17:19:52.622935 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:52.622851 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7894976f45-b8sps" Sep 09 17:19:53.002125 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.002094 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c4798658f-z8qcd"] Sep 09 17:19:53.007680 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:19:53.007634 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f75adb_8242_4ad5_af1b_41f6df199e27.slice/crio-c3d698bce6010251f13cc88b76e8e3fbce51b2f312e583d77c73513ae7273488 WatchSource:0}: Error finding container c3d698bce6010251f13cc88b76e8e3fbce51b2f312e583d77c73513ae7273488: Status 404 returned error can't find the container with id c3d698bce6010251f13cc88b76e8e3fbce51b2f312e583d77c73513ae7273488 Sep 09 17:19:53.017504 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.017201 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zlhb4"] Sep 09 17:19:53.021689 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.021671 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc2sl"] Sep 09 17:19:53.022524 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.022508 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f6l6s"] Sep 09 17:19:53.027747 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:19:53.027487 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9f88fb_c187_4672_bd8e_1916d5f9ff54.slice/crio-dea8e8c67898063c7e0ee4d4ea5caff5203a383b45e62d7f7bb9afc300541678 WatchSource:0}: Error finding container dea8e8c67898063c7e0ee4d4ea5caff5203a383b45e62d7f7bb9afc300541678: Status 404 returned error can't find the container with id dea8e8c67898063c7e0ee4d4ea5caff5203a383b45e62d7f7bb9afc300541678 Sep 09 17:19:53.028222 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.028166 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl"] Sep 09 17:19:53.033142 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:19:53.033112 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda664f0f0_5cc2_421f_bc9f_d35320a559ab.slice/crio-4c3f62254728326dff023de6ee392c0c6791a138d225ffd1023d30857d410b12 WatchSource:0}: Error finding container 4c3f62254728326dff023de6ee392c0c6791a138d225ffd1023d30857d410b12: Status 404 returned error can't find the container with id 4c3f62254728326dff023de6ee392c0c6791a138d225ffd1023d30857d410b12 Sep 09 17:19:53.128189 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.128115 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:53.128189 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.128181 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:53.128610 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.128592 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:53.130303 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.130285 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c3f11a8-d34e-4c31-818d-5bd2d17fe053-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vdjr\" (UID: \"4c3f11a8-d34e-4c31-818d-5bd2d17fe053\") " pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:53.245336 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.245304 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ph5lx\"" Sep 09 17:19:53.253573 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.253553 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2vdjr" Sep 09 17:19:53.530600 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.530568 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2vdjr"] Sep 09 17:19:53.534908 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:19:53.534534 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3f11a8_d34e_4c31_818d_5bd2d17fe053.slice/crio-658410978ed5935ec8670d511cebcf56b72ef09afb2e2e846c2912fa30f1fc07 WatchSource:0}: Error finding container 658410978ed5935ec8670d511cebcf56b72ef09afb2e2e846c2912fa30f1fc07: Status 404 returned error can't find the container with id 658410978ed5935ec8670d511cebcf56b72ef09afb2e2e846c2912fa30f1fc07 Sep 09 17:19:53.694659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.694465 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" event={"ID":"abc4748f-3749-4a07-941f-9ba773ab5e8f","Type":"ContainerStarted","Data":"cc6d5ca1a648b64ddba4b7d09caf3390120c4b6c67047aea3894ac2d1a19a9ae"} Sep 09 17:19:53.694659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.694555 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" event={"ID":"abc4748f-3749-4a07-941f-9ba773ab5e8f","Type":"ContainerStarted","Data":"9c8426426fd2332c128eb8e2400f18b31088f4f4696b0f81b92a7c80161522f1"} Sep 09 17:19:53.694659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.694594 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" event={"ID":"abc4748f-3749-4a07-941f-9ba773ab5e8f","Type":"ContainerStarted","Data":"337b0790b754bdeb5b964d93d6b23c1f0fa682bcb9d908dfcb4b64ddfd11eded"} Sep 09 17:19:53.697918 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.697847 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" event={"ID":"e4f75adb-8242-4ad5-af1b-41f6df199e27","Type":"ContainerStarted","Data":"c3d698bce6010251f13cc88b76e8e3fbce51b2f312e583d77c73513ae7273488"} Sep 09 17:19:53.699701 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.699571 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vdjr" event={"ID":"4c3f11a8-d34e-4c31-818d-5bd2d17fe053","Type":"ContainerStarted","Data":"b2597858c608e2024227560345fc02c46ee9bc61b254363cf04de5d5f7383c5a"} Sep 09 17:19:53.699701 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.699646 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vdjr" event={"ID":"4c3f11a8-d34e-4c31-818d-5bd2d17fe053","Type":"ContainerStarted","Data":"658410978ed5935ec8670d511cebcf56b72ef09afb2e2e846c2912fa30f1fc07"} Sep 09 17:19:53.702024 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.702000 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" event={"ID":"a664f0f0-5cc2-421f-bc9f-d35320a559ab","Type":"ContainerStarted","Data":"4c3f62254728326dff023de6ee392c0c6791a138d225ffd1023d30857d410b12"} Sep 09 17:19:53.703529 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.703499 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f6l6s" event={"ID":"7dc76395-0bd2-439e-b9f9-2c89105cc455","Type":"ContainerStarted","Data":"f66129ccdec71d2b98d5831f360f7ecd45f3a72c4aed47140fdbd994385fcaa9"} Sep 09 17:19:53.704925 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.704900 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc2sl" event={"ID":"8a9f88fb-c187-4672-bd8e-1916d5f9ff54","Type":"ContainerStarted","Data":"dea8e8c67898063c7e0ee4d4ea5caff5203a383b45e62d7f7bb9afc300541678"} Sep 09 17:19:53.706551 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:53.706503 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zlhb4" event={"ID":"22945efa-5c74-4bb9-b708-41a486a7e7c8","Type":"ContainerStarted","Data":"1f5c138be86061d8faf5d440185c9b3159bc6a3f36ee86e5bfbcdd4cb64afd68"} Sep 09 17:19:56.727338 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.727290 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" event={"ID":"efc30c56-e147-45f4-b69c-5d712eb218b1","Type":"ContainerStarted","Data":"809fb12eef9337177546d01e4756ee6bc033b6127b0372480a5db81e0a6970f8"} Sep 09 17:19:56.730365 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.730237 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc2sl" event={"ID":"8a9f88fb-c187-4672-bd8e-1916d5f9ff54","Type":"ContainerStarted","Data":"c5c66e45501d462b18a114cb4a2cb34412ca46bd5c1064a452de465b8386ce2a"} Sep 09 17:19:56.730365 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.730290 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc2sl" event={"ID":"8a9f88fb-c187-4672-bd8e-1916d5f9ff54","Type":"ContainerStarted","Data":"2e9a30388ac1f5797b859c97c9be10131471cf366c2dce1c0ce8a11b87d38df3"} Sep 09 17:19:56.733647 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.733588 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e6b1781-a2cb-491a-a6e0-37fbe045bbde","Type":"ContainerStarted","Data":"69a82afbfcd0dae611e7e1f6c18100921377dce5804069379f05b3cc1e2621f3"} Sep 09 17:19:56.733762 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.733692 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e6b1781-a2cb-491a-a6e0-37fbe045bbde","Type":"ContainerStarted","Data":"62850197c053b5bf42018c2414539a31160ea592ada0aacfe8a81a6bd1445714"} Sep 09 17:19:56.733762 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.733750 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e6b1781-a2cb-491a-a6e0-37fbe045bbde","Type":"ContainerStarted","Data":"45b826dbdca46fd6faa29aa1d3f9cf9dc06b056235b4f16f6da0c65b2277a90d"} Sep 09 17:19:56.738525 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.738439 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" event={"ID":"abc4748f-3749-4a07-941f-9ba773ab5e8f","Type":"ContainerStarted","Data":"509b795be229d426fe1862a002cea9baf5cb827d7fa870677bf801f6673b650e"} Sep 09 17:19:56.738525 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.738508 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" event={"ID":"abc4748f-3749-4a07-941f-9ba773ab5e8f","Type":"ContainerStarted","Data":"62447b14a2a08b65c6c377f94a69da62af6c8d964247fa9a27722369be9eee8a"} Sep 09 17:19:56.738722 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.738550 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" event={"ID":"abc4748f-3749-4a07-941f-9ba773ab5e8f","Type":"ContainerStarted","Data":"234811b93df2e7692333a6c15f62489c2033e23ca8985c71c678432a29a014ae"} Sep 09 17:19:56.738878 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.738860 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:56.741176 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.741152 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" event={"ID":"e4f75adb-8242-4ad5-af1b-41f6df199e27","Type":"ContainerStarted","Data":"409a4d19bea82c651bf0051d698c8813142dd01a738381d707b02e21a9980434"} Sep 09 17:19:56.741261 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.741205 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" event={"ID":"e4f75adb-8242-4ad5-af1b-41f6df199e27","Type":"ContainerStarted","Data":"4e32e78e7a399b1665ec6d159fadb86b089b684e4a27f5d34aa901da8bd6a265"} Sep 09 17:19:56.741261 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.741235 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" event={"ID":"e4f75adb-8242-4ad5-af1b-41f6df199e27","Type":"ContainerStarted","Data":"6b4397a3e1a04f982ee0751e9005cbbac0178aa2da825e0ce94eebae858e725e"} Sep 09 17:19:56.743732 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.743709 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vdjr" event={"ID":"4c3f11a8-d34e-4c31-818d-5bd2d17fe053","Type":"ContainerStarted","Data":"cde08817bd584d4325c9b446a894982b25bf3ea9e89e4b6e5bb4bb342b777c06"} Sep 09 17:19:56.745219 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.745196 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" event={"ID":"a664f0f0-5cc2-421f-bc9f-d35320a559ab","Type":"ContainerStarted","Data":"65906608c2c165813c59b1297bbbfacd8fe2458780001c9384ebc82e11b50bdb"} Sep 09 17:19:56.746008 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.745948 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" Sep 09 17:19:56.749291 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.749128 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" podStartSLOduration=2.15222333 podStartE2EDuration="6.7491117s" totalImagesPullingTime="4.59688837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:56.746284013 +0000 UTC m=+70.986190367" watchObservedRunningTime="2026-09-09 17:19:56.7491117 +0000 UTC m=+70.989018051" Sep 09 17:19:56.751810 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.751777 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" Sep 09 17:19:56.769281 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.769230 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7c4798658f-z8qcd" podStartSLOduration=2.529156938 podStartE2EDuration="5.769210534s" totalImagesPullingTime="3.240053596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:56.766656589 +0000 UTC m=+71.006562943" watchObservedRunningTime="2026-09-09 17:19:56.769210534 +0000 UTC m=+71.009116888" Sep 09 17:19:56.785243 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.785204 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tc2sl" podStartSLOduration=67.569547298 podStartE2EDuration="1m10.785189129s" totalImagesPullingTime="3.215641831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:56.78406295 +0000 UTC m=+71.023969306" watchObservedRunningTime="2026-09-09 17:19:56.785189129 +0000 UTC m=+71.025095482" Sep 09 17:19:56.802184 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.802133 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5x4fl" podStartSLOduration=2.595983438 podStartE2EDuration="5.802118741s" totalImagesPullingTime="3.206135303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:56.800826774 +0000 UTC m=+71.040733128" watchObservedRunningTime="2026-09-09 17:19:56.802118741 +0000 UTC m=+71.042025095" Sep 09 17:19:56.828897 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:56.828101 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" podStartSLOduration=2.177440894 podStartE2EDuration="7.828088372s" totalImagesPullingTime="5.650647478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:19:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:56.827198673 +0000 UTC m=+71.067105030" watchObservedRunningTime="2026-09-09 17:19:56.828088372 +0000 UTC m=+71.067994724" Sep 09 17:19:57.762939 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:57.762905 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8ddf9ddbd-jzljp" Sep 09 17:19:59.763749 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:59.763663 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zlhb4" event={"ID":"22945efa-5c74-4bb9-b708-41a486a7e7c8","Type":"ContainerStarted","Data":"874fb491b7c9d563bed6ba3a2d13b4acee18499e0f8b8a7874689afa1d6bbd3e"} Sep 09 17:19:59.766969 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:59.766941 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e6b1781-a2cb-491a-a6e0-37fbe045bbde","Type":"ContainerStarted","Data":"026634f8d10b32adac88288007c3c19f39cb03e71ef171ec6fcffd1613840cc8"} Sep 09 17:19:59.767093 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:59.767000 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e6b1781-a2cb-491a-a6e0-37fbe045bbde","Type":"ContainerStarted","Data":"65636e9795852f09062ae93a12a5aa0072af0b4a81de57aa67730ac3fa75c046"} Sep 09 17:19:59.767093 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:59.767047 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e6b1781-a2cb-491a-a6e0-37fbe045bbde","Type":"ContainerStarted","Data":"001cce5b45f2f551a8cd70dc7be7ce283b25a7b78e81aac5f2756ac60b41039b"} Sep 09 17:19:59.769029 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:59.769005 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vdjr" event={"ID":"4c3f11a8-d34e-4c31-818d-5bd2d17fe053","Type":"ContainerStarted","Data":"df9edeb50e1395d03c8b033ff7a3c6f315d204ccf9c1ac916924d82919b44b63"} Sep 09 17:19:59.770377 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:59.770357 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f6l6s" event={"ID":"7dc76395-0bd2-439e-b9f9-2c89105cc455","Type":"ContainerStarted","Data":"66bdbb4a99c8524a7c638bcda2429cfd0b02e7b613b055341ef1a8dc6cb70ea6"} Sep 09 17:19:59.770484 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:59.770472 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:19:59.779079 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:59.779025 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zlhb4" podStartSLOduration=67.294899066 podStartE2EDuration="1m13.779008972s" totalImagesPullingTime="6.484109906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:59.778541746 +0000 UTC m=+74.018448101" watchObservedRunningTime="2026-09-09 17:19:59.779008972 +0000 UTC m=+74.018915327" Sep 09 17:19:59.793601 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:59.793565 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f6l6s" podStartSLOduration=67.335708808 podStartE2EDuration="1m13.793554357s" totalImagesPullingTime="6.457845549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:59.79201378 +0000 UTC m=+74.031920133" watchObservedRunningTime="2026-09-09 17:19:59.793554357 +0000 UTC m=+74.033460709" Sep 09 17:19:59.817286 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:59.817232 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.635830372 podStartE2EDuration="12.817221222s" totalImagesPullingTime="6.163024221s" totalInitContainerRuntime="18.366629ms" isStatefulPod=false podCreationTimestamp="2026-09-09 17:19:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:59.815983737 +0000 UTC m=+74.055890113" watchObservedRunningTime="2026-09-09 17:19:59.817221222 +0000 UTC m=+74.057127570" Sep 09 17:19:59.835528 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:19:59.835470 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2vdjr" podStartSLOduration=17.091492823 podStartE2EDuration="22.835454266s" totalImagesPullingTime="5.743961443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:19:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:19:59.835002804 +0000 UTC m=+74.074909158" watchObservedRunningTime="2026-09-09 17:19:59.835454266 +0000 UTC m=+74.075360619" Sep 09 17:20:11.335030 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:20:11.334913 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:20:11.335030 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:20:11.334969 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:20:30.776880 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:20:30.776849 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f6l6s" Sep 09 17:20:31.340423 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:20:31.340389 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:20:31.344142 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:20:31.344119 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84bbb9896d-75tsk" Sep 09 17:23:40.435835 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:40.435798 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz"] Sep 09 17:23:40.437860 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:40.437845 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" Sep 09 17:23:40.440520 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:40.440496 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 09 17:23:40.440652 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:40.440539 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 09 17:23:40.441719 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:40.441692 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-mq7qr\"" Sep 09 17:23:40.454324 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:40.454305 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz"] Sep 09 17:23:40.572633 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:40.572600 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzl89\" (UniqueName: \"kubernetes.io/projected/9616ac32-7a05-4214-a874-262a0d7d137a-kube-api-access-wzl89\") pod \"feast-operator-controller-manager-74cb9876-w4jxz\" (UID: \"9616ac32-7a05-4214-a874-262a0d7d137a\") " pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" Sep 09 17:23:40.673330 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:40.673298 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzl89\" (UniqueName: \"kubernetes.io/projected/9616ac32-7a05-4214-a874-262a0d7d137a-kube-api-access-wzl89\") pod \"feast-operator-controller-manager-74cb9876-w4jxz\" (UID: \"9616ac32-7a05-4214-a874-262a0d7d137a\") " pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" Sep 09 17:23:40.681634 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:40.681601 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzl89\" (UniqueName: \"kubernetes.io/projected/9616ac32-7a05-4214-a874-262a0d7d137a-kube-api-access-wzl89\") pod \"feast-operator-controller-manager-74cb9876-w4jxz\" (UID: \"9616ac32-7a05-4214-a874-262a0d7d137a\") " pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" Sep 09 17:23:40.747611 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:40.747549 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" Sep 09 17:23:41.012361 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:41.012288 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz"] Sep 09 17:23:41.015361 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:23:41.015335 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9616ac32_7a05_4214_a874_262a0d7d137a.slice/crio-764f0731085bc9f8ab06400ded878fc9c4f13b37120c88055e9e99126243a96b WatchSource:0}: Error finding container 764f0731085bc9f8ab06400ded878fc9c4f13b37120c88055e9e99126243a96b: Status 404 returned error can't find the container with id 764f0731085bc9f8ab06400ded878fc9c4f13b37120c88055e9e99126243a96b Sep 09 17:23:41.539839 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:41.539805 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" event={"ID":"9616ac32-7a05-4214-a874-262a0d7d137a","Type":"ContainerStarted","Data":"764f0731085bc9f8ab06400ded878fc9c4f13b37120c88055e9e99126243a96b"} Sep 09 17:23:43.548774 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:43.548688 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" event={"ID":"9616ac32-7a05-4214-a874-262a0d7d137a","Type":"ContainerStarted","Data":"7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b"} Sep 09 17:23:43.549135 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:43.548784 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" Sep 09 17:23:43.564036 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:43.563991 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" podStartSLOduration=1.3694174860000001 podStartE2EDuration="3.563974902s" totalImagesPullingTime="2.194557416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:23:43.563651114 +0000 UTC m=+297.803557468" watchObservedRunningTime="2026-09-09 17:23:43.563974902 +0000 UTC m=+297.803881254" Sep 09 17:23:46.135895 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:46.135845 2553 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 17:23:54.554220 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:23:54.554188 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" Sep 09 17:24:21.658781 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.658745 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v"] Sep 09 17:24:21.661103 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.661087 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.663560 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.663536 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 17:24:21.663715 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.663568 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 17:24:21.663715 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.663608 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 17:24:21.663919 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.663902 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 17:24:21.663997 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.663903 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 09 17:24:21.663997 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.663907 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-krxqc\"" Sep 09 17:24:21.664153 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.664126 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 17:24:21.664208 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.664152 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 17:24:21.670069 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.670049 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v"] Sep 09 17:24:21.770862 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.770823 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-online-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.770862 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.770866 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-feast-data\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.771100 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.770887 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-ui-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.771100 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.771012 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-offline-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.771100 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.771053 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fbhp\" (UniqueName: \"kubernetes.io/projected/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-kube-api-access-5fbhp\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.771228 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.771115 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-registry-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.872138 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.872093 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fbhp\" (UniqueName: \"kubernetes.io/projected/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-kube-api-access-5fbhp\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.872138 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.872143 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-registry-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.872374 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.872179 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-online-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.872374 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.872205 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-feast-data\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.872374 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.872230 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-ui-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.872374 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.872290 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-offline-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.872374 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:24:21.872309 2553 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 09 17:24:21.872374 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:24:21.872371 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-online-tls podName:0432576d-ec2d-44d4-9ea1-e030b9f8fa99 nodeName:}" failed. No retries permitted until 2026-09-09 17:24:22.372352716 +0000 UTC m=+336.612259063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-online-tls") pod "feast-simple-feast-setup-856fcd7b8c-2q88v" (UID: "0432576d-ec2d-44d4-9ea1-e030b9f8fa99") : secret "feast-simple-feast-setup-online-tls" not found Sep 09 17:24:21.872703 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.872599 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-feast-data\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.874645 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.874605 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-ui-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.874819 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.874799 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-registry-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.874858 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.874801 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-offline-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:21.880773 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:21.880749 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fbhp\" (UniqueName: \"kubernetes.io/projected/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-kube-api-access-5fbhp\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:22.374404 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:22.374373 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-online-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:22.376744 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:22.376721 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-online-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-2q88v\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:22.577946 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:22.577917 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:24:22.852609 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:22.852583 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v"] Sep 09 17:24:22.853437 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:24:22.853417 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0432576d_ec2d_44d4_9ea1_e030b9f8fa99.slice/crio-d0c6710b1c166a1944714c0641daa838a698d783f8287cf548c3fc3f544c0381 WatchSource:0}: Error finding container d0c6710b1c166a1944714c0641daa838a698d783f8287cf548c3fc3f544c0381: Status 404 returned error can't find the container with id d0c6710b1c166a1944714c0641daa838a698d783f8287cf548c3fc3f544c0381 Sep 09 17:24:22.855231 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:22.855216 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 17:24:23.691904 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:23.691855 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerStarted","Data":"d0c6710b1c166a1944714c0641daa838a698d783f8287cf548c3fc3f544c0381"} Sep 09 17:24:50.803594 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:50.803559 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerStarted","Data":"7bf8c01cafef3f740e3c046a7bb39c65d9e32ba3f9659f6a09e1bd934edcc7fd"} Sep 09 17:24:55.823185 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:55.823153 2553 generic.go:356] "Generic (PLEG): container finished" podID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerID="7bf8c01cafef3f740e3c046a7bb39c65d9e32ba3f9659f6a09e1bd934edcc7fd" exitCode=0 Sep 09 17:24:55.823599 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:55.823224 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerDied","Data":"7bf8c01cafef3f740e3c046a7bb39c65d9e32ba3f9659f6a09e1bd934edcc7fd"} Sep 09 17:24:56.830062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:24:56.830026 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerStarted","Data":"710804a3fa79838ee2074bde46066972228cc4498a1f9ebc213977aa9ed9292a"} Sep 09 17:25:01.849461 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:01.849375 2553 generic.go:356] "Generic (PLEG): container finished" podID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerID="710804a3fa79838ee2074bde46066972228cc4498a1f9ebc213977aa9ed9292a" exitCode=0 Sep 09 17:25:01.849833 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:01.849453 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerDied","Data":"710804a3fa79838ee2074bde46066972228cc4498a1f9ebc213977aa9ed9292a"} Sep 09 17:25:01.859761 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:01.859739 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 09 17:25:02.859705 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:02.859662 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerStarted","Data":"3b36cff9fc442e68fdad784a496c7d9071671dee1c2e2659a3b6578cbe009d69"} Sep 09 17:25:02.860163 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:02.859732 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerStarted","Data":"256a7af294bff74e5d6cfaff24f1d667ef2e3cdee782fd8d4656c6827d09719b"} Sep 09 17:25:02.860163 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:02.859753 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerStarted","Data":"6573b55f51c62c9841f06541222244e12dd15f42546ab4409522707a584866f4"} Sep 09 17:25:02.860163 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:02.859771 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerStarted","Data":"60f99543a8cd98b8a7cd13443ca493892ed8a57bcfbd96d7c5b0f3ee6582b1cf"} Sep 09 17:25:02.883712 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:02.883171 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podStartSLOduration=14.407919142 podStartE2EDuration="41.883126525s" totalImagesPullingTime="27.475207383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:25:02.881569378 +0000 UTC m=+377.121475734" watchObservedRunningTime="2026-09-09 17:25:02.883126525 +0000 UTC m=+377.123032878" Sep 09 17:25:04.578205 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:04.578176 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:04.578764 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:04.578737 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:04.578882 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:04.578871 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:04.579231 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:04.579220 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:04.580412 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:04.580363 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.16:8443: connect: connection refused" Sep 09 17:25:04.580544 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:04.580464 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.16:8816: connect: connection refused" Sep 09 17:25:04.580544 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:04.580490 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.16:6567/health\": dial tcp 10.134.0.16:6567: connect: connection refused" Sep 09 17:25:04.580544 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:04.580540 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.16:6571: connect: connection refused" Sep 09 17:25:07.578614 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:07.578570 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.16:8816: connect: connection refused" Sep 09 17:25:07.579203 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:07.578570 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.16:8443: connect: connection refused" Sep 09 17:25:07.579317 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:07.578666 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.16:6567/health\": dial tcp 10.134.0.16:6567: connect: connection refused" Sep 09 17:25:07.579317 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:07.578731 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.16:6571: connect: connection refused" Sep 09 17:25:10.579815 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.579773 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:10.580262 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.579886 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:10.580262 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.580066 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:10.580391 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.580310 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:10.580391 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.580349 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:10.580391 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.580369 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:10.580769 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.580749 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:10.580858 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.580801 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:10.580858 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.580833 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:10.585057 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.585036 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:10.902573 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.902501 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:25:10.905764 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:25:10.905741 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:26:14.074107 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.073914 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v"] Sep 09 17:26:14.074694 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.074238 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="registry" containerID="cri-o://60f99543a8cd98b8a7cd13443ca493892ed8a57bcfbd96d7c5b0f3ee6582b1cf" gracePeriod=30 Sep 09 17:26:14.074694 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.074320 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="ui" containerID="cri-o://3b36cff9fc442e68fdad784a496c7d9071671dee1c2e2659a3b6578cbe009d69" gracePeriod=30 Sep 09 17:26:14.074694 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.074305 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="offline" containerID="cri-o://256a7af294bff74e5d6cfaff24f1d667ef2e3cdee782fd8d4656c6827d09719b" gracePeriod=30 Sep 09 17:26:14.074694 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.074305 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="online" containerID="cri-o://6573b55f51c62c9841f06541222244e12dd15f42546ab4409522707a584866f4" gracePeriod=30 Sep 09 17:26:14.463639 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.459592 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7"] Sep 09 17:26:14.467035 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.467006 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.469831 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.469810 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-m2zqz\"" Sep 09 17:26:14.473233 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.473204 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7"] Sep 09 17:26:14.551987 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.551946 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-feast-data\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.551987 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.551988 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-online-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.552196 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.552019 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-ui-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.552196 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.552046 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-registry-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.552196 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.552090 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hpx\" (UniqueName: \"kubernetes.io/projected/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-kube-api-access-r5hpx\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.552196 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.552109 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-offline-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.652896 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.652857 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-ui-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.652896 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.652899 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-registry-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.653147 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.652937 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hpx\" (UniqueName: \"kubernetes.io/projected/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-kube-api-access-r5hpx\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.653147 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.652966 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-offline-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.653147 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.652993 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-feast-data\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.653147 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.653028 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-online-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.653147 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:26:14.653122 2553 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 17:26:14.653393 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:26:14.653218 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-offline-tls podName:bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7 nodeName:}" failed. No retries permitted until 2026-09-09 17:26:15.153195179 +0000 UTC m=+449.393101510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-offline-tls") pod "feast-simple-feast-setup-856fcd7b8c-mhcq7" (UID: "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 17:26:14.653489 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.653446 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-feast-data\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.655570 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.655533 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-registry-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.655570 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.655551 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-online-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.655763 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.655688 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-ui-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:14.660786 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:14.660762 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hpx\" (UniqueName: \"kubernetes.io/projected/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-kube-api-access-r5hpx\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:15.142732 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:15.142655 2553 generic.go:356] "Generic (PLEG): container finished" podID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerID="3b36cff9fc442e68fdad784a496c7d9071671dee1c2e2659a3b6578cbe009d69" exitCode=0 Sep 09 17:26:15.142732 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:15.142681 2553 generic.go:356] "Generic (PLEG): container finished" podID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerID="6573b55f51c62c9841f06541222244e12dd15f42546ab4409522707a584866f4" exitCode=0 Sep 09 17:26:15.143098 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:15.142730 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerDied","Data":"3b36cff9fc442e68fdad784a496c7d9071671dee1c2e2659a3b6578cbe009d69"} Sep 09 17:26:15.143098 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:15.142772 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerDied","Data":"6573b55f51c62c9841f06541222244e12dd15f42546ab4409522707a584866f4"} Sep 09 17:26:15.156982 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:15.156962 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-offline-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:15.159215 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:15.159191 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-offline-tls\") pod \"feast-simple-feast-setup-856fcd7b8c-mhcq7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:15.378948 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:15.378907 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:15.654932 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:15.654905 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7"] Sep 09 17:26:15.656106 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:26:15.656082 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfc0e80a_5b61_44aa_a4a3_04cd14c4d3c7.slice/crio-9049f852530ea8e2aa1259911577953257155e5eb0d51b5e810994627635b534 WatchSource:0}: Error finding container 9049f852530ea8e2aa1259911577953257155e5eb0d51b5e810994627635b534: Status 404 returned error can't find the container with id 9049f852530ea8e2aa1259911577953257155e5eb0d51b5e810994627635b534 Sep 09 17:26:16.148537 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:16.148456 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerStarted","Data":"a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995"} Sep 09 17:26:16.148537 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:16.148498 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerStarted","Data":"9049f852530ea8e2aa1259911577953257155e5eb0d51b5e810994627635b534"} Sep 09 17:26:20.580964 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:20.580920 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.16:8443: connect: connection refused" Sep 09 17:26:20.903682 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:20.903592 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.16:6567/health\": dial tcp 10.134.0.16:6567: connect: connection refused" Sep 09 17:26:21.168866 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:21.168836 2553 generic.go:356] "Generic (PLEG): container finished" podID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerID="a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995" exitCode=0 Sep 09 17:26:21.169054 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:21.168916 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerDied","Data":"a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995"} Sep 09 17:26:22.174167 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:22.174134 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerStarted","Data":"9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be"} Sep 09 17:26:26.862468 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:26:26.862440 2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfc0e80a_5b61_44aa_a4a3_04cd14c4d3c7.slice/crio-conmon-9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be.scope\": RecentStats: unable to find data in memory cache]" Sep 09 17:26:27.196473 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:27.196442 2553 generic.go:356] "Generic (PLEG): container finished" podID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerID="9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be" exitCode=0 Sep 09 17:26:27.196610 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:27.196528 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerDied","Data":"9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be"} Sep 09 17:26:28.206698 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:28.206659 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerStarted","Data":"53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294"} Sep 09 17:26:28.207162 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:28.206726 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerStarted","Data":"a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24"} Sep 09 17:26:28.207162 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:28.206749 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerStarted","Data":"63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da"} Sep 09 17:26:28.207162 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:28.206767 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerStarted","Data":"c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab"} Sep 09 17:26:28.233285 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:28.233219 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podStartSLOduration=14.233197544 podStartE2EDuration="14.233197544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:26:28.231001885 +0000 UTC m=+462.470908277" watchObservedRunningTime="2026-09-09 17:26:28.233197544 +0000 UTC m=+462.473103898" Sep 09 17:26:30.379655 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:30.379439 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:30.379655 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:30.379516 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:30.379655 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:30.379536 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:30.380424 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:30.380373 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:30.381686 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:30.381649 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.17:6571: connect: connection refused" Sep 09 17:26:30.381799 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:30.381719 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.17:6567/health\": dial tcp 10.134.0.17:6567: connect: connection refused" Sep 09 17:26:30.381799 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:30.381656 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.17:8443: connect: connection refused" Sep 09 17:26:30.381936 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:30.381796 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.17:8816: connect: connection refused" Sep 09 17:26:30.581783 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:30.581494 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.16:8443: connect: connection refused" Sep 09 17:26:30.903766 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:30.903720 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.16:6567/health\": dial tcp 10.134.0.16:6567: connect: connection refused" Sep 09 17:26:33.379419 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:33.379363 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.17:8816: connect: connection refused" Sep 09 17:26:33.379940 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:33.379366 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.17:8443: connect: connection refused" Sep 09 17:26:33.379940 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:33.379553 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.17:6571: connect: connection refused" Sep 09 17:26:33.380137 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:33.380091 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.17:6567/health\": dial tcp 10.134.0.17:6567: connect: connection refused" Sep 09 17:26:35.380000 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:35.379965 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:35.380425 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:35.380025 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:35.380425 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:35.380043 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:35.380425 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:35.380058 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:36.380455 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:36.380426 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:36.380871 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:36.380528 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:36.380871 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:36.380756 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:36.381362 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:36.381334 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:36.381466 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:36.381385 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:36.381760 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:36.381741 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:36.385676 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:36.385655 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:37.258449 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:37.258420 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:26:40.580854 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:40.580815 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.16:8443: connect: connection refused" Sep 09 17:26:40.581292 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:40.580954 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:26:40.903002 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:40.902912 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.16:6567/health\": dial tcp 10.134.0.16:6567: connect: connection refused" Sep 09 17:26:40.903167 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:40.903040 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:26:44.291799 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.291769 2553 generic.go:356] "Generic (PLEG): container finished" podID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerID="256a7af294bff74e5d6cfaff24f1d667ef2e3cdee782fd8d4656c6827d09719b" exitCode=137 Sep 09 17:26:44.291799 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.291793 2553 generic.go:356] "Generic (PLEG): container finished" podID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerID="60f99543a8cd98b8a7cd13443ca493892ed8a57bcfbd96d7c5b0f3ee6582b1cf" exitCode=137 Sep 09 17:26:44.292220 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.291841 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerDied","Data":"256a7af294bff74e5d6cfaff24f1d667ef2e3cdee782fd8d4656c6827d09719b"} Sep 09 17:26:44.292220 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.291879 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerDied","Data":"60f99543a8cd98b8a7cd13443ca493892ed8a57bcfbd96d7c5b0f3ee6582b1cf"} Sep 09 17:26:44.722703 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.722678 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:26:44.808771 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.808741 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-online-tls\") pod \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " Sep 09 17:26:44.808930 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.808786 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-feast-data\") pod \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " Sep 09 17:26:44.808930 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.808806 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-offline-tls\") pod \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " Sep 09 17:26:44.808930 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.808842 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-registry-tls\") pod \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " Sep 09 17:26:44.808930 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.808866 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-ui-tls\") pod \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " Sep 09 17:26:44.808930 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.808901 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fbhp\" (UniqueName: \"kubernetes.io/projected/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-kube-api-access-5fbhp\") pod \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\" (UID: \"0432576d-ec2d-44d4-9ea1-e030b9f8fa99\") " Sep 09 17:26:44.809382 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.809327 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-feast-data" (OuterVolumeSpecName: "feast-data") pod "0432576d-ec2d-44d4-9ea1-e030b9f8fa99" (UID: "0432576d-ec2d-44d4-9ea1-e030b9f8fa99"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:26:44.811214 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.811180 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0432576d-ec2d-44d4-9ea1-e030b9f8fa99" (UID: "0432576d-ec2d-44d4-9ea1-e030b9f8fa99"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:26:44.811214 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.811196 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-kube-api-access-5fbhp" (OuterVolumeSpecName: "kube-api-access-5fbhp") pod "0432576d-ec2d-44d4-9ea1-e030b9f8fa99" (UID: "0432576d-ec2d-44d4-9ea1-e030b9f8fa99"). InnerVolumeSpecName "kube-api-access-5fbhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:26:44.811214 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.811209 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-online-tls" (OuterVolumeSpecName: "online-tls") pod "0432576d-ec2d-44d4-9ea1-e030b9f8fa99" (UID: "0432576d-ec2d-44d4-9ea1-e030b9f8fa99"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:26:44.811425 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.811334 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "0432576d-ec2d-44d4-9ea1-e030b9f8fa99" (UID: "0432576d-ec2d-44d4-9ea1-e030b9f8fa99"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:26:44.811425 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.811347 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "0432576d-ec2d-44d4-9ea1-e030b9f8fa99" (UID: "0432576d-ec2d-44d4-9ea1-e030b9f8fa99"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:26:44.909655 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.909565 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:26:44.909655 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.909591 2553 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-ui-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:26:44.909655 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.909599 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fbhp\" (UniqueName: \"kubernetes.io/projected/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-kube-api-access-5fbhp\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:26:44.909655 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.909610 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:26:44.909655 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.909639 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:26:44.909655 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:44.909647 2553 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0432576d-ec2d-44d4-9ea1-e030b9f8fa99-offline-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:26:45.298702 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:45.298669 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" event={"ID":"0432576d-ec2d-44d4-9ea1-e030b9f8fa99","Type":"ContainerDied","Data":"d0c6710b1c166a1944714c0641daa838a698d783f8287cf548c3fc3f544c0381"} Sep 09 17:26:45.299200 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:45.298730 2553 scope.go:117] "RemoveContainer" containerID="3b36cff9fc442e68fdad784a496c7d9071671dee1c2e2659a3b6578cbe009d69" Sep 09 17:26:45.299200 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:45.298679 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v" Sep 09 17:26:45.308573 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:45.308556 2553 scope.go:117] "RemoveContainer" containerID="256a7af294bff74e5d6cfaff24f1d667ef2e3cdee782fd8d4656c6827d09719b" Sep 09 17:26:45.315907 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:45.315883 2553 scope.go:117] "RemoveContainer" containerID="6573b55f51c62c9841f06541222244e12dd15f42546ab4409522707a584866f4" Sep 09 17:26:45.320784 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:45.320684 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v"] Sep 09 17:26:45.324248 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:45.324226 2553 scope.go:117] "RemoveContainer" containerID="60f99543a8cd98b8a7cd13443ca493892ed8a57bcfbd96d7c5b0f3ee6582b1cf" Sep 09 17:26:45.324495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:45.324476 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-2q88v"] Sep 09 17:26:45.331537 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:45.331520 2553 scope.go:117] "RemoveContainer" containerID="710804a3fa79838ee2074bde46066972228cc4498a1f9ebc213977aa9ed9292a" Sep 09 17:26:45.338269 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:45.338253 2553 scope.go:117] "RemoveContainer" containerID="7bf8c01cafef3f740e3c046a7bb39c65d9e32ba3f9659f6a09e1bd934edcc7fd" Sep 09 17:26:46.347658 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:26:46.347612 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" path="/var/lib/kubelet/pods/0432576d-ec2d-44d4-9ea1-e030b9f8fa99/volumes" Sep 09 17:27:39.637561 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637487 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4"] Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637800 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="online" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637812 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="online" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637822 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="offline" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637827 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="offline" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637837 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="feast-apply" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637843 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="feast-apply" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637852 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="feast-init" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637857 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="feast-init" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637864 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="ui" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637869 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="ui" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637883 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="registry" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637888 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="registry" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637931 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="offline" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637941 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="registry" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637948 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="ui" Sep 09 17:27:39.637983 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.637952 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0432576d-ec2d-44d4-9ea1-e030b9f8fa99" containerName="online" Sep 09 17:27:39.641204 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.641183 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.643680 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.643653 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 17:27:39.643798 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.643777 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 17:27:39.643892 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.643797 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-n922m\"" Sep 09 17:27:39.643892 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.643778 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 09 17:27:39.644047 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.644034 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 17:27:39.644097 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.644073 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 17:27:39.644897 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.644879 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 17:27:39.644998 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.644916 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 17:27:39.651559 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.651540 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4"] Sep 09 17:27:39.709409 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.709378 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/514d45fe-2a58-4bec-8a55-0e1cb994a743-feast-data\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.709560 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.709418 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-offline-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.709560 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.709454 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.709560 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.709517 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdcqh\" (UniqueName: \"kubernetes.io/projected/514d45fe-2a58-4bec-8a55-0e1cb994a743-kube-api-access-cdcqh\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.709738 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.709564 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-ui-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.709738 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.709594 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/514d45fe-2a58-4bec-8a55-0e1cb994a743-registry-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.810531 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.810505 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/514d45fe-2a58-4bec-8a55-0e1cb994a743-registry-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.810694 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.810569 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/514d45fe-2a58-4bec-8a55-0e1cb994a743-feast-data\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.810758 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.810692 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-offline-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.810758 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.810736 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.810867 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.810784 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdcqh\" (UniqueName: \"kubernetes.io/projected/514d45fe-2a58-4bec-8a55-0e1cb994a743-kube-api-access-cdcqh\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.810867 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:27:39.810819 2553 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 17:27:39.810969 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:27:39.810880 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls podName:514d45fe-2a58-4bec-8a55-0e1cb994a743 nodeName:}" failed. No retries permitted until 2026-09-09 17:27:40.31085666 +0000 UTC m=+534.550763005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls") pod "feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" (UID: "514d45fe-2a58-4bec-8a55-0e1cb994a743") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 17:27:39.810969 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:27:39.810891 2553 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 17:27:39.810969 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:27:39.810938 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-ui-tls podName:514d45fe-2a58-4bec-8a55-0e1cb994a743 nodeName:}" failed. No retries permitted until 2026-09-09 17:27:40.310923981 +0000 UTC m=+534.550830312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-ui-tls") pod "feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" (UID: "514d45fe-2a58-4bec-8a55-0e1cb994a743") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 17:27:39.810969 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.810818 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-ui-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.811128 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.810975 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/514d45fe-2a58-4bec-8a55-0e1cb994a743-feast-data\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.811208 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.811181 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/514d45fe-2a58-4bec-8a55-0e1cb994a743-registry-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.813082 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.813061 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-offline-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:39.821094 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:39.821068 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdcqh\" (UniqueName: \"kubernetes.io/projected/514d45fe-2a58-4bec-8a55-0e1cb994a743-kube-api-access-cdcqh\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:40.316374 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:40.316341 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:40.316536 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:40.316387 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-ui-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:40.316536 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:27:40.316506 2553 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 17:27:40.316607 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:27:40.316568 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls podName:514d45fe-2a58-4bec-8a55-0e1cb994a743 nodeName:}" failed. No retries permitted until 2026-09-09 17:27:41.316552409 +0000 UTC m=+535.556458744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls") pod "feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" (UID: "514d45fe-2a58-4bec-8a55-0e1cb994a743") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 17:27:40.318828 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:40.318802 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-ui-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:41.325582 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:41.325528 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:41.327994 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:41.327969 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls\") pod \"feast-simple-feast-remote-setup-5fbfd96b65-bwjq4\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:41.450365 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:41.450323 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:27:41.724802 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:41.724774 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4"] Sep 09 17:27:41.726591 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:27:41.726565 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod514d45fe_2a58_4bec_8a55_0e1cb994a743.slice/crio-8ae0b62002844ebe693442271538fd7c99a6e1ad177f1403b4bf3fe61ff259ab WatchSource:0}: Error finding container 8ae0b62002844ebe693442271538fd7c99a6e1ad177f1403b4bf3fe61ff259ab: Status 404 returned error can't find the container with id 8ae0b62002844ebe693442271538fd7c99a6e1ad177f1403b4bf3fe61ff259ab Sep 09 17:27:42.512006 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:42.511967 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerStarted","Data":"89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80"} Sep 09 17:27:42.512371 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:42.512018 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerStarted","Data":"8ae0b62002844ebe693442271538fd7c99a6e1ad177f1403b4bf3fe61ff259ab"} Sep 09 17:27:46.527928 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:46.527898 2553 generic.go:356] "Generic (PLEG): container finished" podID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerID="89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80" exitCode=0 Sep 09 17:27:46.528248 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:46.527946 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerDied","Data":"89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80"} Sep 09 17:27:47.534029 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:27:47.533996 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerStarted","Data":"34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931"} Sep 09 17:28:07.618865 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:07.618828 2553 generic.go:356] "Generic (PLEG): container finished" podID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerID="34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931" exitCode=0 Sep 09 17:28:07.619264 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:07.618902 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerDied","Data":"34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931"} Sep 09 17:28:07.623955 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:07.623936 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 09 17:28:08.627215 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:08.627171 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerStarted","Data":"c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a"} Sep 09 17:28:08.627755 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:08.627230 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerStarted","Data":"67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569"} Sep 09 17:28:08.627755 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:08.627251 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerStarted","Data":"2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613"} Sep 09 17:28:08.651323 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:08.651264 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podStartSLOduration=29.651247403 podStartE2EDuration="29.651247403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:28:08.648786444 +0000 UTC m=+562.888692807" watchObservedRunningTime="2026-09-09 17:28:08.651247403 +0000 UTC m=+562.891153755" Sep 09 17:28:11.450944 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:11.450907 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:28:11.451505 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:11.451472 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:28:11.451655 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:11.451596 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:28:11.451730 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:11.451666 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:28:11.451730 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:11.451688 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:28:11.451730 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:11.451708 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:28:11.452727 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:11.452694 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="online" probeResult="failure" output="Get \"https://10.134.0.18:6567/health\": dial tcp 10.134.0.18:6567: connect: connection refused" Sep 09 17:28:11.452852 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:11.452716 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.18:8443: connect: connection refused" Sep 09 17:28:11.452852 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:11.452696 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.18:8816: connect: connection refused" Sep 09 17:28:14.451546 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:14.451475 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.18:8443: connect: connection refused" Sep 09 17:28:14.452038 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:14.451475 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.18:8816: connect: connection refused" Sep 09 17:28:14.452038 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:14.451502 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="online" probeResult="failure" output="Get \"https://10.134.0.18:6567/health\": dial tcp 10.134.0.18:6567: connect: connection refused" Sep 09 17:28:17.451510 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:17.451471 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:28:17.452020 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:17.451997 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:28:17.452293 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:17.452275 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:28:17.471269 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:17.471243 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:28:17.674574 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:17.674543 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:28:17.691157 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:28:17.691133 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:29:20.195337 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:20.195264 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4"] Sep 09 17:29:20.197862 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:20.195543 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="online" containerID="cri-o://2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613" gracePeriod=30 Sep 09 17:29:20.197862 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:20.195692 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="offline" containerID="cri-o://67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569" gracePeriod=30 Sep 09 17:29:20.197862 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:20.195682 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="ui" containerID="cri-o://c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a" gracePeriod=30 Sep 09 17:29:20.312745 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:20.312711 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7"] Sep 09 17:29:20.313026 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:20.312996 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="registry" containerID="cri-o://c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab" gracePeriod=30 Sep 09 17:29:20.313150 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:20.313016 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="ui" containerID="cri-o://53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294" gracePeriod=30 Sep 09 17:29:20.313310 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:20.313138 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="offline" containerID="cri-o://a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24" gracePeriod=30 Sep 09 17:29:20.313390 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:20.313186 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="online" containerID="cri-o://63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da" gracePeriod=30 Sep 09 17:29:20.913130 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:20.913093 2553 generic.go:356] "Generic (PLEG): container finished" podID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerID="c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a" exitCode=0 Sep 09 17:29:20.913318 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:20.913130 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerDied","Data":"c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a"} Sep 09 17:29:21.919425 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:21.919395 2553 generic.go:356] "Generic (PLEG): container finished" podID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerID="2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613" exitCode=0 Sep 09 17:29:21.919831 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:21.919463 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerDied","Data":"2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613"} Sep 09 17:29:21.922344 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:21.922322 2553 generic.go:356] "Generic (PLEG): container finished" podID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerID="53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294" exitCode=0 Sep 09 17:29:21.922344 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:21.922340 2553 generic.go:356] "Generic (PLEG): container finished" podID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerID="63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da" exitCode=0 Sep 09 17:29:21.922488 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:21.922360 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerDied","Data":"53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294"} Sep 09 17:29:21.922488 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:21.922383 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerDied","Data":"63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da"} Sep 09 17:29:26.382037 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:26.381989 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.17:8443: connect: connection refused" Sep 09 17:29:27.255878 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:27.255837 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.17:6567/health\": dial tcp 10.134.0.17:6567: connect: connection refused" Sep 09 17:29:27.674265 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:27.674218 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.18:8443: connect: connection refused" Sep 09 17:29:27.674614 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:27.674258 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="online" probeResult="failure" output="Get \"https://10.134.0.18:6567/health\": dial tcp 10.134.0.18:6567: connect: connection refused" Sep 09 17:29:36.381735 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:36.381687 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.17:8443: connect: connection refused" Sep 09 17:29:37.256186 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:37.256146 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.17:6567/health\": dial tcp 10.134.0.17:6567: connect: connection refused" Sep 09 17:29:37.674193 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:37.674146 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.18:8443: connect: connection refused" Sep 09 17:29:37.674660 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:37.674156 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="online" probeResult="failure" output="Get \"https://10.134.0.18:6567/health\": dial tcp 10.134.0.18:6567: connect: connection refused" Sep 09 17:29:46.381731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:46.381687 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.17:8443: connect: connection refused" Sep 09 17:29:46.382140 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:46.381841 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:29:47.255369 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:47.255330 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.17:6567/health\": dial tcp 10.134.0.17:6567: connect: connection refused" Sep 09 17:29:47.255535 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:47.255450 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:29:47.674785 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:47.674739 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.18:8443: connect: connection refused" Sep 09 17:29:47.674785 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:47.674778 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="online" probeResult="failure" output="Get \"https://10.134.0.18:6567/health\": dial tcp 10.134.0.18:6567: connect: connection refused" Sep 09 17:29:47.675162 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:47.674882 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:29:47.675162 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:47.674947 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:29:50.846324 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.846300 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:29:50.955902 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.955875 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:29:50.974989 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.974933 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls\") pod \"514d45fe-2a58-4bec-8a55-0e1cb994a743\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " Sep 09 17:29:50.974989 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.974968 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdcqh\" (UniqueName: \"kubernetes.io/projected/514d45fe-2a58-4bec-8a55-0e1cb994a743-kube-api-access-cdcqh\") pod \"514d45fe-2a58-4bec-8a55-0e1cb994a743\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " Sep 09 17:29:50.975154 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.974997 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-offline-tls\") pod \"514d45fe-2a58-4bec-8a55-0e1cb994a743\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " Sep 09 17:29:50.975154 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.975054 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-ui-tls\") pod \"514d45fe-2a58-4bec-8a55-0e1cb994a743\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " Sep 09 17:29:50.975154 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.975091 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/514d45fe-2a58-4bec-8a55-0e1cb994a743-feast-data\") pod \"514d45fe-2a58-4bec-8a55-0e1cb994a743\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " Sep 09 17:29:50.975154 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.975118 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/514d45fe-2a58-4bec-8a55-0e1cb994a743-registry-tls\") pod \"514d45fe-2a58-4bec-8a55-0e1cb994a743\" (UID: \"514d45fe-2a58-4bec-8a55-0e1cb994a743\") " Sep 09 17:29:50.975536 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.975512 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514d45fe-2a58-4bec-8a55-0e1cb994a743-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "514d45fe-2a58-4bec-8a55-0e1cb994a743" (UID: "514d45fe-2a58-4bec-8a55-0e1cb994a743"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:29:50.975656 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.975571 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514d45fe-2a58-4bec-8a55-0e1cb994a743-feast-data" (OuterVolumeSpecName: "feast-data") pod "514d45fe-2a58-4bec-8a55-0e1cb994a743" (UID: "514d45fe-2a58-4bec-8a55-0e1cb994a743"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:29:50.978018 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.977838 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls" (OuterVolumeSpecName: "online-tls") pod "514d45fe-2a58-4bec-8a55-0e1cb994a743" (UID: "514d45fe-2a58-4bec-8a55-0e1cb994a743"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:29:50.978018 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.977948 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514d45fe-2a58-4bec-8a55-0e1cb994a743-kube-api-access-cdcqh" (OuterVolumeSpecName: "kube-api-access-cdcqh") pod "514d45fe-2a58-4bec-8a55-0e1cb994a743" (UID: "514d45fe-2a58-4bec-8a55-0e1cb994a743"). InnerVolumeSpecName "kube-api-access-cdcqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:29:50.978018 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.977955 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "514d45fe-2a58-4bec-8a55-0e1cb994a743" (UID: "514d45fe-2a58-4bec-8a55-0e1cb994a743"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:29:50.978018 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:50.977977 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "514d45fe-2a58-4bec-8a55-0e1cb994a743" (UID: "514d45fe-2a58-4bec-8a55-0e1cb994a743"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:29:51.037541 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.037504 2553 generic.go:356] "Generic (PLEG): container finished" podID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerID="a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24" exitCode=137 Sep 09 17:29:51.037541 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.037530 2553 generic.go:356] "Generic (PLEG): container finished" podID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerID="c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab" exitCode=137 Sep 09 17:29:51.037771 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.037572 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" Sep 09 17:29:51.037771 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.037591 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerDied","Data":"a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24"} Sep 09 17:29:51.037771 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.037663 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerDied","Data":"c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab"} Sep 09 17:29:51.037771 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.037686 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7" event={"ID":"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7","Type":"ContainerDied","Data":"9049f852530ea8e2aa1259911577953257155e5eb0d51b5e810994627635b534"} Sep 09 17:29:51.037771 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.037681 2553 scope.go:117] "RemoveContainer" containerID="53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294" Sep 09 17:29:51.040200 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.040175 2553 generic.go:356] "Generic (PLEG): container finished" podID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerID="67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569" exitCode=137 Sep 09 17:29:51.040304 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.040242 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerDied","Data":"67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569"} Sep 09 17:29:51.040304 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.040275 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" event={"ID":"514d45fe-2a58-4bec-8a55-0e1cb994a743","Type":"ContainerDied","Data":"8ae0b62002844ebe693442271538fd7c99a6e1ad177f1403b4bf3fe61ff259ab"} Sep 09 17:29:51.040304 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.040290 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4" Sep 09 17:29:51.046024 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.046004 2553 scope.go:117] "RemoveContainer" containerID="a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24" Sep 09 17:29:51.054101 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.054081 2553 scope.go:117] "RemoveContainer" containerID="63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da" Sep 09 17:29:51.061555 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.061537 2553 scope.go:117] "RemoveContainer" containerID="c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab" Sep 09 17:29:51.061743 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.061725 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4"] Sep 09 17:29:51.064935 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.064909 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fbfd96b65-bwjq4"] Sep 09 17:29:51.068865 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.068847 2553 scope.go:117] "RemoveContainer" containerID="9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be" Sep 09 17:29:51.075403 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.075383 2553 scope.go:117] "RemoveContainer" containerID="a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995" Sep 09 17:29:51.075588 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.075573 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5hpx\" (UniqueName: \"kubernetes.io/projected/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-kube-api-access-r5hpx\") pod \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " Sep 09 17:29:51.075674 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.075604 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-feast-data\") pod \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " Sep 09 17:29:51.075715 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.075679 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-offline-tls\") pod \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " Sep 09 17:29:51.075715 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.075705 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-registry-tls\") pod \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " Sep 09 17:29:51.075796 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.075762 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-ui-tls\") pod \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " Sep 09 17:29:51.075844 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.075802 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-online-tls\") pod \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\" (UID: \"bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7\") " Sep 09 17:29:51.076310 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.076050 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.076310 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.076071 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdcqh\" (UniqueName: \"kubernetes.io/projected/514d45fe-2a58-4bec-8a55-0e1cb994a743-kube-api-access-cdcqh\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.076310 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.076086 2553 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-offline-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.076310 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.076098 2553 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/514d45fe-2a58-4bec-8a55-0e1cb994a743-ui-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.076310 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.076111 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/514d45fe-2a58-4bec-8a55-0e1cb994a743-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.076310 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.076124 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/514d45fe-2a58-4bec-8a55-0e1cb994a743-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.076310 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.076208 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-feast-data" (OuterVolumeSpecName: "feast-data") pod "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" (UID: "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:29:51.077790 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.077762 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-kube-api-access-r5hpx" (OuterVolumeSpecName: "kube-api-access-r5hpx") pod "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" (UID: "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7"). InnerVolumeSpecName "kube-api-access-r5hpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:29:51.078108 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.078081 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" (UID: "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:29:51.078186 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.078168 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-online-tls" (OuterVolumeSpecName: "online-tls") pod "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" (UID: "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:29:51.078245 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.078216 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" (UID: "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:29:51.078317 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.078302 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" (UID: "bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:29:51.082787 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.082770 2553 scope.go:117] "RemoveContainer" containerID="53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294" Sep 09 17:29:51.083053 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:29:51.083030 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294\": container with ID starting with 53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294 not found: ID does not exist" containerID="53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294" Sep 09 17:29:51.083094 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.083063 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294"} err="failed to get container status \"53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294\": rpc error: code = NotFound desc = could not find container \"53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294\": container with ID starting with 53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294 not found: ID does not exist" Sep 09 17:29:51.083134 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.083094 2553 scope.go:117] "RemoveContainer" containerID="a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24" Sep 09 17:29:51.083354 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:29:51.083335 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24\": container with ID starting with a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24 not found: ID does not exist" containerID="a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24" Sep 09 17:29:51.083424 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.083361 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24"} err="failed to get container status \"a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24\": rpc error: code = NotFound desc = could not find container \"a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24\": container with ID starting with a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24 not found: ID does not exist" Sep 09 17:29:51.083424 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.083383 2553 scope.go:117] "RemoveContainer" containerID="63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da" Sep 09 17:29:51.083655 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:29:51.083638 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da\": container with ID starting with 63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da not found: ID does not exist" containerID="63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da" Sep 09 17:29:51.083700 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.083661 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da"} err="failed to get container status \"63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da\": rpc error: code = NotFound desc = could not find container \"63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da\": container with ID starting with 63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da not found: ID does not exist" Sep 09 17:29:51.083700 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.083677 2553 scope.go:117] "RemoveContainer" containerID="c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab" Sep 09 17:29:51.083932 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:29:51.083913 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab\": container with ID starting with c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab not found: ID does not exist" containerID="c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab" Sep 09 17:29:51.083974 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.083937 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab"} err="failed to get container status \"c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab\": rpc error: code = NotFound desc = could not find container \"c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab\": container with ID starting with c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab not found: ID does not exist" Sep 09 17:29:51.083974 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.083952 2553 scope.go:117] "RemoveContainer" containerID="9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be" Sep 09 17:29:51.084196 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:29:51.084180 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be\": container with ID starting with 9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be not found: ID does not exist" containerID="9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be" Sep 09 17:29:51.084239 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.084208 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be"} err="failed to get container status \"9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be\": rpc error: code = NotFound desc = could not find container \"9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be\": container with ID starting with 9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be not found: ID does not exist" Sep 09 17:29:51.084239 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.084223 2553 scope.go:117] "RemoveContainer" containerID="a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995" Sep 09 17:29:51.084461 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:29:51.084447 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995\": container with ID starting with a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995 not found: ID does not exist" containerID="a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995" Sep 09 17:29:51.084516 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.084463 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995"} err="failed to get container status \"a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995\": rpc error: code = NotFound desc = could not find container \"a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995\": container with ID starting with a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995 not found: ID does not exist" Sep 09 17:29:51.084516 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.084474 2553 scope.go:117] "RemoveContainer" containerID="53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294" Sep 09 17:29:51.084726 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.084709 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294"} err="failed to get container status \"53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294\": rpc error: code = NotFound desc = could not find container \"53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294\": container with ID starting with 53cf6fe2710a3ed431087dbc3bbed31526cca548ac383eb62bcf6f9ae6e75294 not found: ID does not exist" Sep 09 17:29:51.084780 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.084726 2553 scope.go:117] "RemoveContainer" containerID="a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24" Sep 09 17:29:51.084942 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.084925 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24"} err="failed to get container status \"a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24\": rpc error: code = NotFound desc = could not find container \"a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24\": container with ID starting with a797727ae5e65a04a8cd08137c05401814e7d3dd9f2ffff8202cf64c0987cd24 not found: ID does not exist" Sep 09 17:29:51.084984 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.084943 2553 scope.go:117] "RemoveContainer" containerID="63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da" Sep 09 17:29:51.085167 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.085152 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da"} err="failed to get container status \"63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da\": rpc error: code = NotFound desc = could not find container \"63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da\": container with ID starting with 63c2b0e33c4db497b6aa72cfd374d9f77c5dd09b446e3ca12ccbbb17d49d19da not found: ID does not exist" Sep 09 17:29:51.085222 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.085167 2553 scope.go:117] "RemoveContainer" containerID="c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab" Sep 09 17:29:51.085360 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.085345 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab"} err="failed to get container status \"c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab\": rpc error: code = NotFound desc = could not find container \"c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab\": container with ID starting with c4cf2e0be9afc94428ad3df199991d15a911c8e910de4bb2f998da668d32cdab not found: ID does not exist" Sep 09 17:29:51.085401 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.085360 2553 scope.go:117] "RemoveContainer" containerID="9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be" Sep 09 17:29:51.085545 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.085530 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be"} err="failed to get container status \"9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be\": rpc error: code = NotFound desc = could not find container \"9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be\": container with ID starting with 9310afb723450291ba31684a1a692061157e4d7465b885f36c8367b8a81e64be not found: ID does not exist" Sep 09 17:29:51.085590 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.085544 2553 scope.go:117] "RemoveContainer" containerID="a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995" Sep 09 17:29:51.085763 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.085747 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995"} err="failed to get container status \"a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995\": rpc error: code = NotFound desc = could not find container \"a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995\": container with ID starting with a3d4a1d74131311a7f41e78339f300dfde5ffb71035802e075a8b9a3a04ab995 not found: ID does not exist" Sep 09 17:29:51.085807 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.085763 2553 scope.go:117] "RemoveContainer" containerID="c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a" Sep 09 17:29:51.092240 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.092219 2553 scope.go:117] "RemoveContainer" containerID="67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569" Sep 09 17:29:51.098894 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.098880 2553 scope.go:117] "RemoveContainer" containerID="2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613" Sep 09 17:29:51.106803 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.106789 2553 scope.go:117] "RemoveContainer" containerID="34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931" Sep 09 17:29:51.113420 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.113403 2553 scope.go:117] "RemoveContainer" containerID="89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80" Sep 09 17:29:51.120438 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.120423 2553 scope.go:117] "RemoveContainer" containerID="c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a" Sep 09 17:29:51.120698 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:29:51.120674 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a\": container with ID starting with c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a not found: ID does not exist" containerID="c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a" Sep 09 17:29:51.120757 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.120703 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a"} err="failed to get container status \"c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a\": rpc error: code = NotFound desc = could not find container \"c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a\": container with ID starting with c8d984164ba62a3c37109b8217ac1b3edeaf7f07fcb1dc4866945bf818a9700a not found: ID does not exist" Sep 09 17:29:51.120757 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.120717 2553 scope.go:117] "RemoveContainer" containerID="67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569" Sep 09 17:29:51.120954 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:29:51.120939 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569\": container with ID starting with 67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569 not found: ID does not exist" containerID="67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569" Sep 09 17:29:51.120991 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.120958 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569"} err="failed to get container status \"67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569\": rpc error: code = NotFound desc = could not find container \"67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569\": container with ID starting with 67e73595137c6e53a7f44c4645e39841cd0208fddf4eba6c2bf4494e9bc88569 not found: ID does not exist" Sep 09 17:29:51.120991 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.120971 2553 scope.go:117] "RemoveContainer" containerID="2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613" Sep 09 17:29:51.121197 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:29:51.121177 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613\": container with ID starting with 2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613 not found: ID does not exist" containerID="2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613" Sep 09 17:29:51.121261 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.121207 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613"} err="failed to get container status \"2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613\": rpc error: code = NotFound desc = could not find container \"2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613\": container with ID starting with 2987b1dcfa9ed9f8fd969c0245ca9cc38a3179110da33b3c313a5ec80d562613 not found: ID does not exist" Sep 09 17:29:51.121261 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.121227 2553 scope.go:117] "RemoveContainer" containerID="34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931" Sep 09 17:29:51.121459 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:29:51.121445 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931\": container with ID starting with 34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931 not found: ID does not exist" containerID="34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931" Sep 09 17:29:51.121537 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.121461 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931"} err="failed to get container status \"34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931\": rpc error: code = NotFound desc = could not find container \"34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931\": container with ID starting with 34e5beae43dac278e763b25d5f415a922de319cf69ffd888833fb79b6e52d931 not found: ID does not exist" Sep 09 17:29:51.121537 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.121474 2553 scope.go:117] "RemoveContainer" containerID="89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80" Sep 09 17:29:51.121717 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:29:51.121701 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80\": container with ID starting with 89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80 not found: ID does not exist" containerID="89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80" Sep 09 17:29:51.121766 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.121725 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80"} err="failed to get container status \"89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80\": rpc error: code = NotFound desc = could not find container \"89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80\": container with ID starting with 89014d5f2768d1cdc5b9db226002e0825db15f98937015e9147c438986f84f80 not found: ID does not exist" Sep 09 17:29:51.177136 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.177115 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.177207 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.177139 2553 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-offline-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.177207 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.177150 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.177207 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.177159 2553 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-ui-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.177207 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.177167 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.177207 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.177174 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5hpx\" (UniqueName: \"kubernetes.io/projected/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7-kube-api-access-r5hpx\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:29:51.359085 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.359059 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7"] Sep 09 17:29:51.363190 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:51.363165 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-856fcd7b8c-mhcq7"] Sep 09 17:29:52.346014 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:52.345980 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" path="/var/lib/kubelet/pods/514d45fe-2a58-4bec-8a55-0e1cb994a743/volumes" Sep 09 17:29:52.346507 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:29:52.346493 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" path="/var/lib/kubelet/pods/bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7/volumes" Sep 09 17:30:03.123143 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123109 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-vbpfd"] Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123515 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="online" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123533 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="online" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123545 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="feast-apply" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123554 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="feast-apply" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123565 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="offline" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123574 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="offline" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123582 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="offline" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123590 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="offline" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123607 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="ui" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123615 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="ui" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123652 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="feast-apply" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123661 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="feast-apply" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123675 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="ui" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123682 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="ui" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123691 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="feast-init" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123699 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="feast-init" Sep 09 17:30:03.123716 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123724 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="registry" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123732 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="registry" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123744 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="feast-init" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123751 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="feast-init" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123760 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="online" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123768 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="online" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123843 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="ui" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123855 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="online" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123864 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="offline" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123874 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="offline" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123883 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfc0e80a-5b61-44aa-a4a3-04cd14c4d3c7" containerName="registry" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123891 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="online" Sep 09 17:30:03.124495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.123900 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="514d45fe-2a58-4bec-8a55-0e1cb994a743" containerName="ui" Sep 09 17:30:03.128816 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.128797 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-vbpfd" Sep 09 17:30:03.131445 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.131421 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 09 17:30:03.131564 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.131480 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 17:30:03.131564 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.131487 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-ccxdx\"" Sep 09 17:30:03.132797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.132779 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 17:30:03.135864 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.135841 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-vbpfd"] Sep 09 17:30:03.154424 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.154398 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-4f6db"] Sep 09 17:30:03.157643 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.157608 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4f6db" Sep 09 17:30:03.166992 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.166969 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4f6db"] Sep 09 17:30:03.264390 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.264354 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxj5h\" (UniqueName: \"kubernetes.io/projected/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-kube-api-access-bxj5h\") pod \"postgres-864846d4f5-vbpfd\" (UID: \"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2\") " pod="test-ns-feast/postgres-864846d4f5-vbpfd" Sep 09 17:30:03.264553 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.264423 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-postgresdata\") pod \"postgres-864846d4f5-vbpfd\" (UID: \"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2\") " pod="test-ns-feast/postgres-864846d4f5-vbpfd" Sep 09 17:30:03.264553 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.264443 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbf26\" (UniqueName: \"kubernetes.io/projected/5402425a-2fde-42dc-85af-4de131be19f2-kube-api-access-sbf26\") pod \"redis-69bbd798b6-4f6db\" (UID: \"5402425a-2fde-42dc-85af-4de131be19f2\") " pod="test-ns-feast/redis-69bbd798b6-4f6db" Sep 09 17:30:03.365133 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.365095 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-postgresdata\") pod \"postgres-864846d4f5-vbpfd\" (UID: \"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2\") " pod="test-ns-feast/postgres-864846d4f5-vbpfd" Sep 09 17:30:03.365133 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.365130 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbf26\" (UniqueName: \"kubernetes.io/projected/5402425a-2fde-42dc-85af-4de131be19f2-kube-api-access-sbf26\") pod \"redis-69bbd798b6-4f6db\" (UID: \"5402425a-2fde-42dc-85af-4de131be19f2\") " pod="test-ns-feast/redis-69bbd798b6-4f6db" Sep 09 17:30:03.365321 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.365160 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxj5h\" (UniqueName: \"kubernetes.io/projected/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-kube-api-access-bxj5h\") pod \"postgres-864846d4f5-vbpfd\" (UID: \"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2\") " pod="test-ns-feast/postgres-864846d4f5-vbpfd" Sep 09 17:30:03.365539 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.365518 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-postgresdata\") pod \"postgres-864846d4f5-vbpfd\" (UID: \"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2\") " pod="test-ns-feast/postgres-864846d4f5-vbpfd" Sep 09 17:30:03.372973 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.372949 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxj5h\" (UniqueName: \"kubernetes.io/projected/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-kube-api-access-bxj5h\") pod \"postgres-864846d4f5-vbpfd\" (UID: \"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2\") " pod="test-ns-feast/postgres-864846d4f5-vbpfd" Sep 09 17:30:03.373085 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.372949 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbf26\" (UniqueName: \"kubernetes.io/projected/5402425a-2fde-42dc-85af-4de131be19f2-kube-api-access-sbf26\") pod \"redis-69bbd798b6-4f6db\" (UID: \"5402425a-2fde-42dc-85af-4de131be19f2\") " pod="test-ns-feast/redis-69bbd798b6-4f6db" Sep 09 17:30:03.451083 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.451050 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-vbpfd" Sep 09 17:30:03.466498 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.466470 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4f6db" Sep 09 17:30:03.719795 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.719715 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-vbpfd"] Sep 09 17:30:03.722554 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:30:03.722530 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80e6a8a1_1691_4d08_a6b6_b8bc366b37c2.slice/crio-eeaa1ce3128681752a523759724265095c40dccf59149606ab6c78f800c79d9e WatchSource:0}: Error finding container eeaa1ce3128681752a523759724265095c40dccf59149606ab6c78f800c79d9e: Status 404 returned error can't find the container with id eeaa1ce3128681752a523759724265095c40dccf59149606ab6c78f800c79d9e Sep 09 17:30:03.724272 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.724256 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 17:30:03.740941 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:03.740915 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4f6db"] Sep 09 17:30:03.743406 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:30:03.743384 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5402425a_2fde_42dc_85af_4de131be19f2.slice/crio-bf04d00cd2f44ae0bb660670c9bb35f7d9765243a2da5cd3b3ea3f5c091554e2 WatchSource:0}: Error finding container bf04d00cd2f44ae0bb660670c9bb35f7d9765243a2da5cd3b3ea3f5c091554e2: Status 404 returned error can't find the container with id bf04d00cd2f44ae0bb660670c9bb35f7d9765243a2da5cd3b3ea3f5c091554e2 Sep 09 17:30:04.087830 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:04.087745 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4f6db" event={"ID":"5402425a-2fde-42dc-85af-4de131be19f2","Type":"ContainerStarted","Data":"bf04d00cd2f44ae0bb660670c9bb35f7d9765243a2da5cd3b3ea3f5c091554e2"} Sep 09 17:30:04.088800 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:04.088775 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-vbpfd" event={"ID":"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2","Type":"ContainerStarted","Data":"eeaa1ce3128681752a523759724265095c40dccf59149606ab6c78f800c79d9e"} Sep 09 17:30:07.107356 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:07.107315 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4f6db" event={"ID":"5402425a-2fde-42dc-85af-4de131be19f2","Type":"ContainerStarted","Data":"0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335"} Sep 09 17:30:07.123430 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:07.123386 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-4f6db" podStartSLOduration=0.95709948 podStartE2EDuration="4.123371155s" totalImagesPullingTime="3.166271675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:30:07.121674516 +0000 UTC m=+681.361580869" watchObservedRunningTime="2026-09-09 17:30:07.123371155 +0000 UTC m=+681.363277508" Sep 09 17:30:10.121789 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:10.121746 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-vbpfd" event={"ID":"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2","Type":"ContainerStarted","Data":"f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99"} Sep 09 17:30:14.014866 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.014812 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-vbpfd" podStartSLOduration=5.291254869 podStartE2EDuration="11.01479437s" totalImagesPullingTime="5.723539501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:30:10.135745655 +0000 UTC m=+684.375652008" watchObservedRunningTime="2026-09-09 17:30:14.01479437 +0000 UTC m=+688.254700723" Sep 09 17:30:14.015227 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.015163 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w"] Sep 09 17:30:14.021864 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.021838 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.024703 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.024680 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 09 17:30:14.024836 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.024718 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 09 17:30:14.024836 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.024731 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 09 17:30:14.024836 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.024722 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-hrkvz\"" Sep 09 17:30:14.024971 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.024947 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 09 17:30:14.032854 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.032831 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w"] Sep 09 17:30:14.051110 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.051085 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6abebb87-be82-4c08-9f20-04a540b9b63b-feast-data\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.051212 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.051115 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.051212 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.051149 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-online-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.051286 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.051243 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-registry-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.051286 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.051275 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq4bd\" (UniqueName: \"kubernetes.io/projected/6abebb87-be82-4c08-9f20-04a540b9b63b-kube-api-access-nq4bd\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.151654 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.151610 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-online-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.151778 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.151677 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-registry-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.151778 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.151702 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4bd\" (UniqueName: \"kubernetes.io/projected/6abebb87-be82-4c08-9f20-04a540b9b63b-kube-api-access-nq4bd\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.151778 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.151738 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6abebb87-be82-4c08-9f20-04a540b9b63b-feast-data\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.151778 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.151756 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.151931 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:30:14.151776 2553 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Sep 09 17:30:14.151931 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:30:14.151839 2553 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Sep 09 17:30:14.151931 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:30:14.151853 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-online-tls podName:6abebb87-be82-4c08-9f20-04a540b9b63b nodeName:}" failed. No retries permitted until 2026-09-09 17:30:14.651836623 +0000 UTC m=+688.891742973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-online-tls") pod "feast-credit-scoring-77ddff596d-fdp9w" (UID: "6abebb87-be82-4c08-9f20-04a540b9b63b") : secret "feast-credit-scoring-online-tls" not found Sep 09 17:30:14.151931 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:30:14.151888 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls podName:6abebb87-be82-4c08-9f20-04a540b9b63b nodeName:}" failed. No retries permitted until 2026-09-09 17:30:14.651875548 +0000 UTC m=+688.891781879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls") pod "feast-credit-scoring-77ddff596d-fdp9w" (UID: "6abebb87-be82-4c08-9f20-04a540b9b63b") : secret "feast-credit-scoring-offline-tls" not found Sep 09 17:30:14.152153 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.152136 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6abebb87-be82-4c08-9f20-04a540b9b63b-feast-data\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.154103 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.154083 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-registry-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.160745 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.160726 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq4bd\" (UniqueName: \"kubernetes.io/projected/6abebb87-be82-4c08-9f20-04a540b9b63b-kube-api-access-nq4bd\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.656885 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.656841 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.657063 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.656924 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-online-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:14.657063 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:30:14.656970 2553 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Sep 09 17:30:14.657063 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:30:14.657038 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls podName:6abebb87-be82-4c08-9f20-04a540b9b63b nodeName:}" failed. No retries permitted until 2026-09-09 17:30:15.657020769 +0000 UTC m=+689.896927116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls") pod "feast-credit-scoring-77ddff596d-fdp9w" (UID: "6abebb87-be82-4c08-9f20-04a540b9b63b") : secret "feast-credit-scoring-offline-tls" not found Sep 09 17:30:14.659265 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:14.659242 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-online-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:15.665353 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:15.665321 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:15.667664 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:15.667639 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls\") pod \"feast-credit-scoring-77ddff596d-fdp9w\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:15.833565 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:15.833530 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:16.109020 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:16.108996 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w"] Sep 09 17:30:16.110776 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:30:16.110748 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abebb87_be82_4c08_9f20_04a540b9b63b.slice/crio-4a255d48224b72df692bb939a878ef15089c4be45272bd9bde882d0de2fc294b WatchSource:0}: Error finding container 4a255d48224b72df692bb939a878ef15089c4be45272bd9bde882d0de2fc294b: Status 404 returned error can't find the container with id 4a255d48224b72df692bb939a878ef15089c4be45272bd9bde882d0de2fc294b Sep 09 17:30:16.144492 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:16.144464 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" event={"ID":"6abebb87-be82-4c08-9f20-04a540b9b63b","Type":"ContainerStarted","Data":"4a255d48224b72df692bb939a878ef15089c4be45272bd9bde882d0de2fc294b"} Sep 09 17:30:17.148980 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:17.148952 2553 generic.go:356] "Generic (PLEG): container finished" podID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerID="f7eb27882ffadc6065bed63be58841fdff39d43b84d66348613aa6e943fe0a32" exitCode=0 Sep 09 17:30:17.149357 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:17.149014 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" event={"ID":"6abebb87-be82-4c08-9f20-04a540b9b63b","Type":"ContainerDied","Data":"f7eb27882ffadc6065bed63be58841fdff39d43b84d66348613aa6e943fe0a32"} Sep 09 17:30:18.155411 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:18.155375 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" event={"ID":"6abebb87-be82-4c08-9f20-04a540b9b63b","Type":"ContainerStarted","Data":"ca500b4f45d78cd4fe8d5ebff915e2d5bcb22b6d9c0fe8fe9779a4b440060340"} Sep 09 17:30:24.179727 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:24.179695 2553 generic.go:356] "Generic (PLEG): container finished" podID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerID="ca500b4f45d78cd4fe8d5ebff915e2d5bcb22b6d9c0fe8fe9779a4b440060340" exitCode=0 Sep 09 17:30:24.180107 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:24.179765 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" event={"ID":"6abebb87-be82-4c08-9f20-04a540b9b63b","Type":"ContainerDied","Data":"ca500b4f45d78cd4fe8d5ebff915e2d5bcb22b6d9c0fe8fe9779a4b440060340"} Sep 09 17:30:25.187695 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:25.187660 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" event={"ID":"6abebb87-be82-4c08-9f20-04a540b9b63b","Type":"ContainerStarted","Data":"20aacf622110c0ffa7050ecedd4770fcca0b903c8ce4cd708ee861cf9ccec4c5"} Sep 09 17:30:25.188173 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:25.187723 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" event={"ID":"6abebb87-be82-4c08-9f20-04a540b9b63b","Type":"ContainerStarted","Data":"ba54cb70af25776244668b70abd0ce2618a0db3b8cb5f3eb1e7a6be199ccd04d"} Sep 09 17:30:25.188173 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:25.187742 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" event={"ID":"6abebb87-be82-4c08-9f20-04a540b9b63b","Type":"ContainerStarted","Data":"6018163e0ff1439511bce8e094ec3d308f72d4a2bc42fbaadd292bbed82106cc"} Sep 09 17:30:25.212688 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:25.212613 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podStartSLOduration=12.212593445 podStartE2EDuration="12.212593445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:30:25.210476819 +0000 UTC m=+699.450383171" watchObservedRunningTime="2026-09-09 17:30:25.212593445 +0000 UTC m=+699.452499799" Sep 09 17:30:25.834700 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:25.834658 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:25.834887 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:25.834776 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:25.834887 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:25.834807 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:27.834171 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:27.834089 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:27.834652 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:27.834232 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:27.834652 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:27.834262 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:27.835726 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:27.835694 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.21:6571: connect: connection refused" Sep 09 17:30:27.835945 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:27.835711 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.21:8816: connect: connection refused" Sep 09 17:30:27.836023 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:27.835730 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Sep 09 17:30:30.834731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:30.834678 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.21:8816: connect: connection refused" Sep 09 17:30:30.834731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:30.834684 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.21:6571: connect: connection refused" Sep 09 17:30:30.835277 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:30.834751 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Sep 09 17:30:33.834450 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:33.834393 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.21:6571: connect: connection refused" Sep 09 17:30:33.835412 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:33.835385 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:33.836411 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:33.836388 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:33.841160 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:33.841138 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:34.249813 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:34.249784 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:36.834482 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:36.834435 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.21:6571: connect: connection refused" Sep 09 17:30:39.834754 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:39.834678 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:30:39.835504 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:30:39.835472 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:31:44.464381 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.464350 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w"] Sep 09 17:31:44.464858 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.464572 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="registry" containerID="cri-o://6018163e0ff1439511bce8e094ec3d308f72d4a2bc42fbaadd292bbed82106cc" gracePeriod=30 Sep 09 17:31:44.464858 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.464657 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="offline" containerID="cri-o://20aacf622110c0ffa7050ecedd4770fcca0b903c8ce4cd708ee861cf9ccec4c5" gracePeriod=30 Sep 09 17:31:44.464858 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.464646 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="online" containerID="cri-o://ba54cb70af25776244668b70abd0ce2618a0db3b8cb5f3eb1e7a6be199ccd04d" gracePeriod=30 Sep 09 17:31:44.481274 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.481246 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-vbpfd"] Sep 09 17:31:44.481430 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.481402 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-vbpfd" podUID="80e6a8a1-1691-4d08-a6b6-b8bc366b37c2" containerName="postgres" containerID="cri-o://f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99" gracePeriod=30 Sep 09 17:31:44.491259 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.491234 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4f6db"] Sep 09 17:31:44.491410 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.491390 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-4f6db" podUID="5402425a-2fde-42dc-85af-4de131be19f2" containerName="redis" containerID="cri-o://0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335" gracePeriod=30 Sep 09 17:31:44.745006 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.744984 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-vbpfd" Sep 09 17:31:44.747834 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.747817 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4f6db" Sep 09 17:31:44.893249 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.893215 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-postgresdata\") pod \"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2\" (UID: \"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2\") " Sep 09 17:31:44.893407 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.893281 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbf26\" (UniqueName: \"kubernetes.io/projected/5402425a-2fde-42dc-85af-4de131be19f2-kube-api-access-sbf26\") pod \"5402425a-2fde-42dc-85af-4de131be19f2\" (UID: \"5402425a-2fde-42dc-85af-4de131be19f2\") " Sep 09 17:31:44.893407 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.893315 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxj5h\" (UniqueName: \"kubernetes.io/projected/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-kube-api-access-bxj5h\") pod \"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2\" (UID: \"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2\") " Sep 09 17:31:44.895421 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.895386 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-kube-api-access-bxj5h" (OuterVolumeSpecName: "kube-api-access-bxj5h") pod "80e6a8a1-1691-4d08-a6b6-b8bc366b37c2" (UID: "80e6a8a1-1691-4d08-a6b6-b8bc366b37c2"). InnerVolumeSpecName "kube-api-access-bxj5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:31:44.895545 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.895449 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5402425a-2fde-42dc-85af-4de131be19f2-kube-api-access-sbf26" (OuterVolumeSpecName: "kube-api-access-sbf26") pod "5402425a-2fde-42dc-85af-4de131be19f2" (UID: "5402425a-2fde-42dc-85af-4de131be19f2"). InnerVolumeSpecName "kube-api-access-sbf26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:31:44.923297 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.923260 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "80e6a8a1-1691-4d08-a6b6-b8bc366b37c2" (UID: "80e6a8a1-1691-4d08-a6b6-b8bc366b37c2"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:31:44.994048 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.993964 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxj5h\" (UniqueName: \"kubernetes.io/projected/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-kube-api-access-bxj5h\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:31:44.994048 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.993998 2553 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2-postgresdata\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:31:44.994048 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:44.994012 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbf26\" (UniqueName: \"kubernetes.io/projected/5402425a-2fde-42dc-85af-4de131be19f2-kube-api-access-sbf26\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:31:45.535674 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.535645 2553 generic.go:356] "Generic (PLEG): container finished" podID="5402425a-2fde-42dc-85af-4de131be19f2" containerID="0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335" exitCode=0 Sep 09 17:31:45.536036 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.535681 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4f6db" Sep 09 17:31:45.536036 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.535731 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4f6db" event={"ID":"5402425a-2fde-42dc-85af-4de131be19f2","Type":"ContainerDied","Data":"0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335"} Sep 09 17:31:45.536036 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.535774 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4f6db" event={"ID":"5402425a-2fde-42dc-85af-4de131be19f2","Type":"ContainerDied","Data":"bf04d00cd2f44ae0bb660670c9bb35f7d9765243a2da5cd3b3ea3f5c091554e2"} Sep 09 17:31:45.536036 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.535793 2553 scope.go:117] "RemoveContainer" containerID="0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335" Sep 09 17:31:45.538680 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.538653 2553 generic.go:356] "Generic (PLEG): container finished" podID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerID="ba54cb70af25776244668b70abd0ce2618a0db3b8cb5f3eb1e7a6be199ccd04d" exitCode=0 Sep 09 17:31:45.538777 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.538724 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" event={"ID":"6abebb87-be82-4c08-9f20-04a540b9b63b","Type":"ContainerDied","Data":"ba54cb70af25776244668b70abd0ce2618a0db3b8cb5f3eb1e7a6be199ccd04d"} Sep 09 17:31:45.540188 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.540138 2553 generic.go:356] "Generic (PLEG): container finished" podID="80e6a8a1-1691-4d08-a6b6-b8bc366b37c2" containerID="f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99" exitCode=0 Sep 09 17:31:45.540263 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.540202 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-vbpfd" event={"ID":"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2","Type":"ContainerDied","Data":"f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99"} Sep 09 17:31:45.540263 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.540241 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-vbpfd" event={"ID":"80e6a8a1-1691-4d08-a6b6-b8bc366b37c2","Type":"ContainerDied","Data":"eeaa1ce3128681752a523759724265095c40dccf59149606ab6c78f800c79d9e"} Sep 09 17:31:45.540331 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.540300 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-vbpfd" Sep 09 17:31:45.544594 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.544553 2553 scope.go:117] "RemoveContainer" containerID="0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335" Sep 09 17:31:45.544927 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:31:45.544902 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335\": container with ID starting with 0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335 not found: ID does not exist" containerID="0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335" Sep 09 17:31:45.545019 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.544932 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335"} err="failed to get container status \"0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335\": rpc error: code = NotFound desc = could not find container \"0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335\": container with ID starting with 0dbe2d51e9fe963adc33decd8d45c5d58530d943057751d8aaaca9fb69cc7335 not found: ID does not exist" Sep 09 17:31:45.545019 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.544949 2553 scope.go:117] "RemoveContainer" containerID="f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99" Sep 09 17:31:45.552480 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.552465 2553 scope.go:117] "RemoveContainer" containerID="f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99" Sep 09 17:31:45.552752 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:31:45.552734 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99\": container with ID starting with f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99 not found: ID does not exist" containerID="f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99" Sep 09 17:31:45.552820 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.552758 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99"} err="failed to get container status \"f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99\": rpc error: code = NotFound desc = could not find container \"f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99\": container with ID starting with f0668f6fb1cf27e427debe10855e60366f593dda545b606c57201b857f29eb99 not found: ID does not exist" Sep 09 17:31:45.557098 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.557078 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4f6db"] Sep 09 17:31:45.561374 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.561352 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4f6db"] Sep 09 17:31:45.574338 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.574314 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-vbpfd"] Sep 09 17:31:45.576264 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:45.576244 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-vbpfd"] Sep 09 17:31:46.346175 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:46.346134 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5402425a-2fde-42dc-85af-4de131be19f2" path="/var/lib/kubelet/pods/5402425a-2fde-42dc-85af-4de131be19f2/volumes" Sep 09 17:31:46.346477 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:46.346462 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e6a8a1-1691-4d08-a6b6-b8bc366b37c2" path="/var/lib/kubelet/pods/80e6a8a1-1691-4d08-a6b6-b8bc366b37c2/volumes" Sep 09 17:31:54.247699 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:31:54.247660 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Sep 09 17:32:04.247255 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:04.247208 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Sep 09 17:32:14.247649 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:14.247538 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Sep 09 17:32:14.247993 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:14.247684 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:32:14.648130 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:14.648100 2553 generic.go:356] "Generic (PLEG): container finished" podID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerID="20aacf622110c0ffa7050ecedd4770fcca0b903c8ce4cd708ee861cf9ccec4c5" exitCode=137 Sep 09 17:32:14.648130 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:14.648125 2553 generic.go:356] "Generic (PLEG): container finished" podID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerID="6018163e0ff1439511bce8e094ec3d308f72d4a2bc42fbaadd292bbed82106cc" exitCode=137 Sep 09 17:32:14.648351 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:14.648172 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" event={"ID":"6abebb87-be82-4c08-9f20-04a540b9b63b","Type":"ContainerDied","Data":"20aacf622110c0ffa7050ecedd4770fcca0b903c8ce4cd708ee861cf9ccec4c5"} Sep 09 17:32:14.648351 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:14.648232 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" event={"ID":"6abebb87-be82-4c08-9f20-04a540b9b63b","Type":"ContainerDied","Data":"6018163e0ff1439511bce8e094ec3d308f72d4a2bc42fbaadd292bbed82106cc"} Sep 09 17:32:15.103665 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.103642 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:32:15.120415 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.120385 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-registry-tls\") pod \"6abebb87-be82-4c08-9f20-04a540b9b63b\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " Sep 09 17:32:15.120415 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.120416 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq4bd\" (UniqueName: \"kubernetes.io/projected/6abebb87-be82-4c08-9f20-04a540b9b63b-kube-api-access-nq4bd\") pod \"6abebb87-be82-4c08-9f20-04a540b9b63b\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " Sep 09 17:32:15.120555 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.120456 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6abebb87-be82-4c08-9f20-04a540b9b63b-feast-data\") pod \"6abebb87-be82-4c08-9f20-04a540b9b63b\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " Sep 09 17:32:15.120555 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.120500 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-online-tls\") pod \"6abebb87-be82-4c08-9f20-04a540b9b63b\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " Sep 09 17:32:15.120555 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.120514 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls\") pod \"6abebb87-be82-4c08-9f20-04a540b9b63b\" (UID: \"6abebb87-be82-4c08-9f20-04a540b9b63b\") " Sep 09 17:32:15.122537 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.122510 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abebb87-be82-4c08-9f20-04a540b9b63b-kube-api-access-nq4bd" (OuterVolumeSpecName: "kube-api-access-nq4bd") pod "6abebb87-be82-4c08-9f20-04a540b9b63b" (UID: "6abebb87-be82-4c08-9f20-04a540b9b63b"). InnerVolumeSpecName "kube-api-access-nq4bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:32:15.122670 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.122556 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6abebb87-be82-4c08-9f20-04a540b9b63b" (UID: "6abebb87-be82-4c08-9f20-04a540b9b63b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:32:15.122670 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.122573 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-online-tls" (OuterVolumeSpecName: "online-tls") pod "6abebb87-be82-4c08-9f20-04a540b9b63b" (UID: "6abebb87-be82-4c08-9f20-04a540b9b63b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:32:15.122847 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.122824 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "6abebb87-be82-4c08-9f20-04a540b9b63b" (UID: "6abebb87-be82-4c08-9f20-04a540b9b63b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:32:15.129254 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.129234 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abebb87-be82-4c08-9f20-04a540b9b63b-feast-data" (OuterVolumeSpecName: "feast-data") pod "6abebb87-be82-4c08-9f20-04a540b9b63b" (UID: "6abebb87-be82-4c08-9f20-04a540b9b63b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:32:15.221997 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.221970 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6abebb87-be82-4c08-9f20-04a540b9b63b-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:32:15.221997 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.221993 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:32:15.221997 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.222003 2553 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-offline-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:32:15.222187 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.222010 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6abebb87-be82-4c08-9f20-04a540b9b63b-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:32:15.222187 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.222019 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq4bd\" (UniqueName: \"kubernetes.io/projected/6abebb87-be82-4c08-9f20-04a540b9b63b-kube-api-access-nq4bd\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:32:15.654738 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.654706 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" event={"ID":"6abebb87-be82-4c08-9f20-04a540b9b63b","Type":"ContainerDied","Data":"4a255d48224b72df692bb939a878ef15089c4be45272bd9bde882d0de2fc294b"} Sep 09 17:32:15.654738 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.654729 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w" Sep 09 17:32:15.655186 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.654770 2553 scope.go:117] "RemoveContainer" containerID="20aacf622110c0ffa7050ecedd4770fcca0b903c8ce4cd708ee861cf9ccec4c5" Sep 09 17:32:15.663171 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.663152 2553 scope.go:117] "RemoveContainer" containerID="ba54cb70af25776244668b70abd0ce2618a0db3b8cb5f3eb1e7a6be199ccd04d" Sep 09 17:32:15.670224 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.670208 2553 scope.go:117] "RemoveContainer" containerID="6018163e0ff1439511bce8e094ec3d308f72d4a2bc42fbaadd292bbed82106cc" Sep 09 17:32:15.677045 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.676951 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w"] Sep 09 17:32:15.677327 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.677311 2553 scope.go:117] "RemoveContainer" containerID="ca500b4f45d78cd4fe8d5ebff915e2d5bcb22b6d9c0fe8fe9779a4b440060340" Sep 09 17:32:15.681043 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.681019 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-77ddff596d-fdp9w"] Sep 09 17:32:15.684321 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:15.684301 2553 scope.go:117] "RemoveContainer" containerID="f7eb27882ffadc6065bed63be58841fdff39d43b84d66348613aa6e943fe0a32" Sep 09 17:32:16.346842 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:32:16.346802 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" path="/var/lib/kubelet/pods/6abebb87-be82-4c08-9f20-04a540b9b63b/volumes" Sep 09 17:35:09.330665 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330579 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wsv27"] Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330905 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="feast-apply" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330919 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="feast-apply" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330928 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="registry" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330934 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="registry" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330946 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="online" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330951 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="online" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330967 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="feast-init" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330976 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="feast-init" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330988 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="offline" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330993 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="offline" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.330999 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e6a8a1-1691-4d08-a6b6-b8bc366b37c2" containerName="postgres" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.331004 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e6a8a1-1691-4d08-a6b6-b8bc366b37c2" containerName="postgres" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.331010 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5402425a-2fde-42dc-85af-4de131be19f2" containerName="redis" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.331016 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5402425a-2fde-42dc-85af-4de131be19f2" containerName="redis" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.331068 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="registry" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.331076 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="offline" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.331082 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e6a8a1-1691-4d08-a6b6-b8bc366b37c2" containerName="postgres" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.331088 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5402425a-2fde-42dc-85af-4de131be19f2" containerName="redis" Sep 09 17:35:09.332922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.331092 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6abebb87-be82-4c08-9f20-04a540b9b63b" containerName="online" Sep 09 17:35:09.333755 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.333740 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" Sep 09 17:35:09.336445 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.336402 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 09 17:35:09.336574 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.336459 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-tdrvq\"" Sep 09 17:35:09.337813 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.337796 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 09 17:35:09.348945 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.348893 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wsv27"] Sep 09 17:35:09.366075 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.366032 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl899\" (UniqueName: \"kubernetes.io/projected/e6e65988-3098-4628-85b1-ef4c05caaf6f-kube-api-access-zl899\") pod \"redis-69bbd798b6-wsv27\" (UID: \"e6e65988-3098-4628-85b1-ef4c05caaf6f\") " pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" Sep 09 17:35:09.466887 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.466849 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl899\" (UniqueName: \"kubernetes.io/projected/e6e65988-3098-4628-85b1-ef4c05caaf6f-kube-api-access-zl899\") pod \"redis-69bbd798b6-wsv27\" (UID: \"e6e65988-3098-4628-85b1-ef4c05caaf6f\") " pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" Sep 09 17:35:09.474965 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.474934 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl899\" (UniqueName: \"kubernetes.io/projected/e6e65988-3098-4628-85b1-ef4c05caaf6f-kube-api-access-zl899\") pod \"redis-69bbd798b6-wsv27\" (UID: \"e6e65988-3098-4628-85b1-ef4c05caaf6f\") " pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" Sep 09 17:35:09.650176 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.650086 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" Sep 09 17:35:09.917433 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:09.917400 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wsv27"] Sep 09 17:35:09.926683 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:35:09.926654 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6e65988_3098_4628_85b1_ef4c05caaf6f.slice/crio-cc62d6bfee4a8e0cba105c5158a45a7e5a7c90fc0f3ef1a050fcd6bc82e6127b WatchSource:0}: Error finding container cc62d6bfee4a8e0cba105c5158a45a7e5a7c90fc0f3ef1a050fcd6bc82e6127b: Status 404 returned error can't find the container with id cc62d6bfee4a8e0cba105c5158a45a7e5a7c90fc0f3ef1a050fcd6bc82e6127b Sep 09 17:35:10.287836 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:10.287748 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" event={"ID":"e6e65988-3098-4628-85b1-ef4c05caaf6f","Type":"ContainerStarted","Data":"8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c"} Sep 09 17:35:10.287836 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:10.287790 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" event={"ID":"e6e65988-3098-4628-85b1-ef4c05caaf6f","Type":"ContainerStarted","Data":"cc62d6bfee4a8e0cba105c5158a45a7e5a7c90fc0f3ef1a050fcd6bc82e6127b"} Sep 09 17:35:10.302339 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:10.302293 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" podStartSLOduration=1.302278027 podStartE2EDuration="1.302278027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:10.301605272 +0000 UTC m=+984.541511627" watchObservedRunningTime="2026-09-09 17:35:10.302278027 +0000 UTC m=+984.542184380" Sep 09 17:35:14.773736 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:14.773693 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5jj5j"] Sep 09 17:35:14.776732 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:14.776714 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" Sep 09 17:35:14.779241 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:14.779216 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 09 17:35:14.783245 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:14.783221 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5jj5j"] Sep 09 17:35:14.808211 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:14.808187 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b9765c4b-2903-40b3-8f60-3b6621be1068-postgresdata\") pod \"postgres-864846d4f5-5jj5j\" (UID: \"b9765c4b-2903-40b3-8f60-3b6621be1068\") " pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" Sep 09 17:35:14.808366 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:14.808230 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xwz\" (UniqueName: \"kubernetes.io/projected/b9765c4b-2903-40b3-8f60-3b6621be1068-kube-api-access-p4xwz\") pod \"postgres-864846d4f5-5jj5j\" (UID: \"b9765c4b-2903-40b3-8f60-3b6621be1068\") " pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" Sep 09 17:35:14.908677 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:14.908648 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xwz\" (UniqueName: \"kubernetes.io/projected/b9765c4b-2903-40b3-8f60-3b6621be1068-kube-api-access-p4xwz\") pod \"postgres-864846d4f5-5jj5j\" (UID: \"b9765c4b-2903-40b3-8f60-3b6621be1068\") " pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" Sep 09 17:35:14.908811 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:14.908721 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b9765c4b-2903-40b3-8f60-3b6621be1068-postgresdata\") pod \"postgres-864846d4f5-5jj5j\" (UID: \"b9765c4b-2903-40b3-8f60-3b6621be1068\") " pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" Sep 09 17:35:14.909019 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:14.909005 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b9765c4b-2903-40b3-8f60-3b6621be1068-postgresdata\") pod \"postgres-864846d4f5-5jj5j\" (UID: \"b9765c4b-2903-40b3-8f60-3b6621be1068\") " pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" Sep 09 17:35:14.917556 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:14.917529 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xwz\" (UniqueName: \"kubernetes.io/projected/b9765c4b-2903-40b3-8f60-3b6621be1068-kube-api-access-p4xwz\") pod \"postgres-864846d4f5-5jj5j\" (UID: \"b9765c4b-2903-40b3-8f60-3b6621be1068\") " pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" Sep 09 17:35:15.086479 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:15.086400 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" Sep 09 17:35:15.354361 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:15.354288 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5jj5j"] Sep 09 17:35:15.357303 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:35:15.357271 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9765c4b_2903_40b3_8f60_3b6621be1068.slice/crio-1325907f2908798cdc7ce34fa9f671e8d9ae4c2f2ce6df6ce5b1add66e73d0e5 WatchSource:0}: Error finding container 1325907f2908798cdc7ce34fa9f671e8d9ae4c2f2ce6df6ce5b1add66e73d0e5: Status 404 returned error can't find the container with id 1325907f2908798cdc7ce34fa9f671e8d9ae4c2f2ce6df6ce5b1add66e73d0e5 Sep 09 17:35:16.310984 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:16.310946 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" event={"ID":"b9765c4b-2903-40b3-8f60-3b6621be1068","Type":"ContainerStarted","Data":"e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c"} Sep 09 17:35:16.311362 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:16.310993 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" event={"ID":"b9765c4b-2903-40b3-8f60-3b6621be1068","Type":"ContainerStarted","Data":"1325907f2908798cdc7ce34fa9f671e8d9ae4c2f2ce6df6ce5b1add66e73d0e5"} Sep 09 17:35:16.326148 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:16.326101 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" podStartSLOduration=2.326080124 podStartE2EDuration="2.326080124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:16.325012645 +0000 UTC m=+990.564918999" watchObservedRunningTime="2026-09-09 17:35:16.326080124 +0000 UTC m=+990.565986477" Sep 09 17:35:20.378989 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.378956 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr"] Sep 09 17:35:20.382279 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.382259 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.384672 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.384650 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 09 17:35:20.384956 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.384934 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 09 17:35:20.385038 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.384993 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 09 17:35:20.385100 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.385077 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-thb9b\"" Sep 09 17:35:20.392919 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.392897 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr"] Sep 09 17:35:20.449903 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.449878 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-registry-tls\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.449903 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.449905 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-feast-data\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.450029 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.449934 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjbt\" (UniqueName: \"kubernetes.io/projected/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-kube-api-access-xhjbt\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.450066 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.450048 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-online-tls\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.550890 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.550863 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjbt\" (UniqueName: \"kubernetes.io/projected/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-kube-api-access-xhjbt\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.550986 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.550910 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-online-tls\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.550986 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.550949 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-registry-tls\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.550986 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.550971 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-feast-data\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.551385 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.551365 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-feast-data\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.553241 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.553222 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-online-tls\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.553359 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.553340 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-registry-tls\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.559296 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.559272 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjbt\" (UniqueName: \"kubernetes.io/projected/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-kube-api-access-xhjbt\") pod \"feast-credit-scoring-b5d96f645-lpvjr\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.691503 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.691473 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:20.966156 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:20.966129 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr"] Sep 09 17:35:20.967594 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:35:20.967564 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b59d9a4_6405_4b01_bbdc_5dfa67bfff18.slice/crio-80249c5caab7c0b99cc4a7e37e6917bf7855fb729ece68bdedd3c0a3c3c0373b WatchSource:0}: Error finding container 80249c5caab7c0b99cc4a7e37e6917bf7855fb729ece68bdedd3c0a3c3c0373b: Status 404 returned error can't find the container with id 80249c5caab7c0b99cc4a7e37e6917bf7855fb729ece68bdedd3c0a3c3c0373b Sep 09 17:35:21.329974 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:21.329895 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" event={"ID":"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18","Type":"ContainerStarted","Data":"cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a"} Sep 09 17:35:21.329974 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:21.329944 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" event={"ID":"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18","Type":"ContainerStarted","Data":"80249c5caab7c0b99cc4a7e37e6917bf7855fb729ece68bdedd3c0a3c3c0373b"} Sep 09 17:35:22.335321 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:22.335231 2553 generic.go:356] "Generic (PLEG): container finished" podID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerID="cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a" exitCode=0 Sep 09 17:35:22.335321 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:22.335290 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" event={"ID":"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18","Type":"ContainerDied","Data":"cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a"} Sep 09 17:35:23.341351 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:23.341315 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" event={"ID":"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18","Type":"ContainerStarted","Data":"0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2"} Sep 09 17:35:29.366309 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:29.366273 2553 generic.go:356] "Generic (PLEG): container finished" podID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerID="0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2" exitCode=0 Sep 09 17:35:29.366728 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:29.366325 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" event={"ID":"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18","Type":"ContainerDied","Data":"0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2"} Sep 09 17:35:30.373288 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:30.373248 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" event={"ID":"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18","Type":"ContainerStarted","Data":"44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747"} Sep 09 17:35:30.373671 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:30.373316 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" event={"ID":"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18","Type":"ContainerStarted","Data":"5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c"} Sep 09 17:35:30.397141 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:30.397094 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" podStartSLOduration=10.397079759 podStartE2EDuration="10.397079759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:30.394584896 +0000 UTC m=+1004.634491249" watchObservedRunningTime="2026-09-09 17:35:30.397079759 +0000 UTC m=+1004.636986112" Sep 09 17:35:30.692260 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:30.692227 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:30.692403 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:30.692302 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:32.692118 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:32.692083 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:32.692497 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:32.692142 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:32.693783 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:32.693752 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.24:6571: connect: connection refused" Sep 09 17:35:32.693839 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:32.693771 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 17:35:35.692217 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:35.692172 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.24:6571: connect: connection refused" Sep 09 17:35:35.698126 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:35.698093 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:35.701552 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:35.701529 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:38.692538 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:38.692495 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.24:6571: connect: connection refused" Sep 09 17:35:41.692488 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:41.692458 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:41.693256 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:41.693235 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:35:46.272070 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.272039 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65"] Sep 09 17:35:46.275540 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.275523 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.278020 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.277995 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 09 17:35:46.278148 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.278060 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 09 17:35:46.278148 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.278087 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 09 17:35:46.278148 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.278112 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-fvrlh\"" Sep 09 17:35:46.290227 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.290205 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65"] Sep 09 17:35:46.357894 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.357869 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-online-tls\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.357998 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.357910 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh24q\" (UniqueName: \"kubernetes.io/projected/7d486c5c-b5b8-4868-99c2-537b9b8921b1-kube-api-access-xh24q\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.357998 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.357966 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7d486c5c-b5b8-4868-99c2-537b9b8921b1-feast-data\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.358073 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.357997 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-registry-tls\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.458611 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.458585 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-online-tls\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.458749 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.458651 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh24q\" (UniqueName: \"kubernetes.io/projected/7d486c5c-b5b8-4868-99c2-537b9b8921b1-kube-api-access-xh24q\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.458749 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.458688 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7d486c5c-b5b8-4868-99c2-537b9b8921b1-feast-data\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.458749 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.458706 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-registry-tls\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.458915 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:35:46.458790 2553 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found Sep 09 17:35:46.458915 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:35:46.458840 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-registry-tls podName:7d486c5c-b5b8-4868-99c2-537b9b8921b1 nodeName:}" failed. No retries permitted until 2026-09-09 17:35:46.958824962 +0000 UTC m=+1021.198731292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-registry-tls") pod "feast-driver-ranking-66c846fd6-5bm65" (UID: "7d486c5c-b5b8-4868-99c2-537b9b8921b1") : secret "feast-driver-ranking-registry-tls" not found Sep 09 17:35:46.459124 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.459106 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7d486c5c-b5b8-4868-99c2-537b9b8921b1-feast-data\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.460822 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.460801 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-online-tls\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.467510 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.467491 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh24q\" (UniqueName: \"kubernetes.io/projected/7d486c5c-b5b8-4868-99c2-537b9b8921b1-kube-api-access-xh24q\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.963848 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.963810 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-registry-tls\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:46.966319 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:46.966294 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-registry-tls\") pod \"feast-driver-ranking-66c846fd6-5bm65\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:47.190544 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:47.190509 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:35:47.463464 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:47.463445 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65"] Sep 09 17:35:47.465484 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:35:47.465451 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d486c5c_b5b8_4868_99c2_537b9b8921b1.slice/crio-f3e0c3101e443bef2668529a18a68da216242670f8d2b2ed430a2c15b9fc39bc WatchSource:0}: Error finding container f3e0c3101e443bef2668529a18a68da216242670f8d2b2ed430a2c15b9fc39bc: Status 404 returned error can't find the container with id f3e0c3101e443bef2668529a18a68da216242670f8d2b2ed430a2c15b9fc39bc Sep 09 17:35:48.445015 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:48.444978 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" event={"ID":"7d486c5c-b5b8-4868-99c2-537b9b8921b1","Type":"ContainerStarted","Data":"1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d"} Sep 09 17:35:48.445183 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:48.445037 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" event={"ID":"7d486c5c-b5b8-4868-99c2-537b9b8921b1","Type":"ContainerStarted","Data":"f3e0c3101e443bef2668529a18a68da216242670f8d2b2ed430a2c15b9fc39bc"} Sep 09 17:35:52.464094 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:52.464056 2553 generic.go:356] "Generic (PLEG): container finished" podID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerID="1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d" exitCode=0 Sep 09 17:35:52.464469 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:52.464132 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" event={"ID":"7d486c5c-b5b8-4868-99c2-537b9b8921b1","Type":"ContainerDied","Data":"1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d"} Sep 09 17:35:53.470517 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:53.470482 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" event={"ID":"7d486c5c-b5b8-4868-99c2-537b9b8921b1","Type":"ContainerStarted","Data":"9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a"} Sep 09 17:35:59.495702 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:59.495634 2553 generic.go:356] "Generic (PLEG): container finished" podID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerID="9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a" exitCode=0 Sep 09 17:35:59.495702 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:35:59.495657 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" event={"ID":"7d486c5c-b5b8-4868-99c2-537b9b8921b1","Type":"ContainerDied","Data":"9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a"} Sep 09 17:36:00.503979 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:00.503946 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" event={"ID":"7d486c5c-b5b8-4868-99c2-537b9b8921b1","Type":"ContainerStarted","Data":"feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266"} Sep 09 17:36:00.504488 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:00.504000 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" event={"ID":"7d486c5c-b5b8-4868-99c2-537b9b8921b1","Type":"ContainerStarted","Data":"c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592"} Sep 09 17:36:00.528218 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:00.527870 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" podStartSLOduration=14.527854531 podStartE2EDuration="14.527854531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:36:00.527077073 +0000 UTC m=+1034.766983429" watchObservedRunningTime="2026-09-09 17:36:00.527854531 +0000 UTC m=+1034.767760885" Sep 09 17:36:02.191501 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:02.191466 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:36:02.192005 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:02.191534 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:36:02.193314 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:02.193279 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 09 17:36:02.193463 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:02.193307 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 09 17:36:05.190863 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:05.190815 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 09 17:36:05.191300 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:05.190834 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 09 17:36:07.191519 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:07.191470 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:36:07.191930 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:07.191546 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:36:08.191386 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:08.191331 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 09 17:36:08.197273 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:08.197242 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:36:08.200960 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:08.200931 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:36:11.191507 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:11.191475 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:36:11.192247 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:36:11.192230 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:37:28.873579 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:28.873550 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz"] Sep 09 17:37:28.874071 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:28.873796 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" podUID="9616ac32-7a05-4214-a874-262a0d7d137a" containerName="manager" containerID="cri-o://7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b" gracePeriod=10 Sep 09 17:37:29.133505 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.133482 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" Sep 09 17:37:29.145375 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.145349 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr"] Sep 09 17:37:29.145604 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.145578 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="registry" containerID="cri-o://5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c" gracePeriod=30 Sep 09 17:37:29.145729 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.145654 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="online" containerID="cri-o://44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747" gracePeriod=30 Sep 09 17:37:29.151599 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.151541 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65"] Sep 09 17:37:29.151728 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.151645 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzl89\" (UniqueName: \"kubernetes.io/projected/9616ac32-7a05-4214-a874-262a0d7d137a-kube-api-access-wzl89\") pod \"9616ac32-7a05-4214-a874-262a0d7d137a\" (UID: \"9616ac32-7a05-4214-a874-262a0d7d137a\") " Sep 09 17:37:29.151976 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.151926 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="registry" containerID="cri-o://c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592" gracePeriod=30 Sep 09 17:37:29.151976 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.151959 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="online" containerID="cri-o://feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266" gracePeriod=30 Sep 09 17:37:29.154968 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.154867 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9616ac32-7a05-4214-a874-262a0d7d137a-kube-api-access-wzl89" (OuterVolumeSpecName: "kube-api-access-wzl89") pod "9616ac32-7a05-4214-a874-262a0d7d137a" (UID: "9616ac32-7a05-4214-a874-262a0d7d137a"). InnerVolumeSpecName "kube-api-access-wzl89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:37:29.252923 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.252895 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzl89\" (UniqueName: \"kubernetes.io/projected/9616ac32-7a05-4214-a874-262a0d7d137a-kube-api-access-wzl89\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:29.864186 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.864151 2553 generic.go:356] "Generic (PLEG): container finished" podID="9616ac32-7a05-4214-a874-262a0d7d137a" containerID="7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b" exitCode=0 Sep 09 17:37:29.864389 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.864230 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" Sep 09 17:37:29.864389 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.864236 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" event={"ID":"9616ac32-7a05-4214-a874-262a0d7d137a","Type":"ContainerDied","Data":"7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b"} Sep 09 17:37:29.864389 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.864302 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz" event={"ID":"9616ac32-7a05-4214-a874-262a0d7d137a","Type":"ContainerDied","Data":"764f0731085bc9f8ab06400ded878fc9c4f13b37120c88055e9e99126243a96b"} Sep 09 17:37:29.864389 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.864331 2553 scope.go:117] "RemoveContainer" containerID="7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b" Sep 09 17:37:29.880667 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.880527 2553 scope.go:117] "RemoveContainer" containerID="7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b" Sep 09 17:37:29.880921 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:37:29.880869 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b\": container with ID starting with 7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b not found: ID does not exist" containerID="7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b" Sep 09 17:37:29.880921 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.880899 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b"} err="failed to get container status \"7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b\": rpc error: code = NotFound desc = could not find container \"7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b\": container with ID starting with 7d48f9fdac120b70e1353ece9eefe3605a1237f182ef49de303eda5931b11f2b not found: ID does not exist" Sep 09 17:37:29.891404 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.891382 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz"] Sep 09 17:37:29.894137 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:29.894116 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-74cb9876-w4jxz"] Sep 09 17:37:30.347115 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:30.347079 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9616ac32-7a05-4214-a874-262a0d7d137a" path="/var/lib/kubelet/pods/9616ac32-7a05-4214-a874-262a0d7d137a/volumes" Sep 09 17:37:30.871027 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:30.870998 2553 generic.go:356] "Generic (PLEG): container finished" podID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerID="44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747" exitCode=0 Sep 09 17:37:30.871214 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:30.871072 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" event={"ID":"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18","Type":"ContainerDied","Data":"44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747"} Sep 09 17:37:30.873392 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:30.873369 2553 generic.go:356] "Generic (PLEG): container finished" podID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerID="feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266" exitCode=0 Sep 09 17:37:30.873519 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:30.873439 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" event={"ID":"7d486c5c-b5b8-4868-99c2-537b9b8921b1","Type":"ContainerDied","Data":"feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266"} Sep 09 17:37:31.324393 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.324358 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5jj5j"] Sep 09 17:37:31.324851 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.324553 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" podUID="b9765c4b-2903-40b3-8f60-3b6621be1068" containerName="postgres" containerID="cri-o://e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c" gracePeriod=30 Sep 09 17:37:31.329093 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.329068 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wsv27"] Sep 09 17:37:31.331656 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.329247 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" podUID="e6e65988-3098-4628-85b1-ef4c05caaf6f" containerName="redis" containerID="cri-o://8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c" gracePeriod=30 Sep 09 17:37:31.585577 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.585554 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" Sep 09 17:37:31.588354 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.588334 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" Sep 09 17:37:31.672681 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.672649 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b9765c4b-2903-40b3-8f60-3b6621be1068-postgresdata\") pod \"b9765c4b-2903-40b3-8f60-3b6621be1068\" (UID: \"b9765c4b-2903-40b3-8f60-3b6621be1068\") " Sep 09 17:37:31.672834 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.672717 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4xwz\" (UniqueName: \"kubernetes.io/projected/b9765c4b-2903-40b3-8f60-3b6621be1068-kube-api-access-p4xwz\") pod \"b9765c4b-2903-40b3-8f60-3b6621be1068\" (UID: \"b9765c4b-2903-40b3-8f60-3b6621be1068\") " Sep 09 17:37:31.672834 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.672773 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl899\" (UniqueName: \"kubernetes.io/projected/e6e65988-3098-4628-85b1-ef4c05caaf6f-kube-api-access-zl899\") pod \"e6e65988-3098-4628-85b1-ef4c05caaf6f\" (UID: \"e6e65988-3098-4628-85b1-ef4c05caaf6f\") " Sep 09 17:37:31.674815 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.674789 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9765c4b-2903-40b3-8f60-3b6621be1068-kube-api-access-p4xwz" (OuterVolumeSpecName: "kube-api-access-p4xwz") pod "b9765c4b-2903-40b3-8f60-3b6621be1068" (UID: "b9765c4b-2903-40b3-8f60-3b6621be1068"). InnerVolumeSpecName "kube-api-access-p4xwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:37:31.674929 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.674814 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e65988-3098-4628-85b1-ef4c05caaf6f-kube-api-access-zl899" (OuterVolumeSpecName: "kube-api-access-zl899") pod "e6e65988-3098-4628-85b1-ef4c05caaf6f" (UID: "e6e65988-3098-4628-85b1-ef4c05caaf6f"). InnerVolumeSpecName "kube-api-access-zl899". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:37:31.702287 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.702261 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9765c4b-2903-40b3-8f60-3b6621be1068-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "b9765c4b-2903-40b3-8f60-3b6621be1068" (UID: "b9765c4b-2903-40b3-8f60-3b6621be1068"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:37:31.774076 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.774055 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4xwz\" (UniqueName: \"kubernetes.io/projected/b9765c4b-2903-40b3-8f60-3b6621be1068-kube-api-access-p4xwz\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:31.774076 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.774074 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl899\" (UniqueName: \"kubernetes.io/projected/e6e65988-3098-4628-85b1-ef4c05caaf6f-kube-api-access-zl899\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:31.774209 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.774084 2553 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b9765c4b-2903-40b3-8f60-3b6621be1068-postgresdata\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:31.878262 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.878203 2553 generic.go:356] "Generic (PLEG): container finished" podID="e6e65988-3098-4628-85b1-ef4c05caaf6f" containerID="8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c" exitCode=0 Sep 09 17:37:31.878262 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.878243 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" Sep 09 17:37:31.878262 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.878250 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" event={"ID":"e6e65988-3098-4628-85b1-ef4c05caaf6f","Type":"ContainerDied","Data":"8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c"} Sep 09 17:37:31.878419 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.878284 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-wsv27" event={"ID":"e6e65988-3098-4628-85b1-ef4c05caaf6f","Type":"ContainerDied","Data":"cc62d6bfee4a8e0cba105c5158a45a7e5a7c90fc0f3ef1a050fcd6bc82e6127b"} Sep 09 17:37:31.878419 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.878306 2553 scope.go:117] "RemoveContainer" containerID="8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c" Sep 09 17:37:31.879638 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.879601 2553 generic.go:356] "Generic (PLEG): container finished" podID="b9765c4b-2903-40b3-8f60-3b6621be1068" containerID="e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c" exitCode=0 Sep 09 17:37:31.879724 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.879648 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" event={"ID":"b9765c4b-2903-40b3-8f60-3b6621be1068","Type":"ContainerDied","Data":"e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c"} Sep 09 17:37:31.879724 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.879666 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" Sep 09 17:37:31.879724 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.879688 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-5jj5j" event={"ID":"b9765c4b-2903-40b3-8f60-3b6621be1068","Type":"ContainerDied","Data":"1325907f2908798cdc7ce34fa9f671e8d9ae4c2f2ce6df6ce5b1add66e73d0e5"} Sep 09 17:37:31.887416 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.887401 2553 scope.go:117] "RemoveContainer" containerID="8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c" Sep 09 17:37:31.887716 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:37:31.887700 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c\": container with ID starting with 8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c not found: ID does not exist" containerID="8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c" Sep 09 17:37:31.887767 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.887725 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c"} err="failed to get container status \"8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c\": rpc error: code = NotFound desc = could not find container \"8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c\": container with ID starting with 8aedff1b18aa4ccebe022ed3aa3984b6606f70ef728eb74afddf20b4b187758c not found: ID does not exist" Sep 09 17:37:31.887767 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.887739 2553 scope.go:117] "RemoveContainer" containerID="e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c" Sep 09 17:37:31.894851 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.894835 2553 scope.go:117] "RemoveContainer" containerID="e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c" Sep 09 17:37:31.895078 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:37:31.895061 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c\": container with ID starting with e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c not found: ID does not exist" containerID="e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c" Sep 09 17:37:31.895123 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.895084 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c"} err="failed to get container status \"e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c\": rpc error: code = NotFound desc = could not find container \"e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c\": container with ID starting with e70a6db8c816fdf0415b50cf7f8392c91c980672bc222bad3d62c33a1fae242c not found: ID does not exist" Sep 09 17:37:31.898252 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.898233 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wsv27"] Sep 09 17:37:31.901733 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.901713 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wsv27"] Sep 09 17:37:31.911796 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.911776 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5jj5j"] Sep 09 17:37:31.915199 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:31.915175 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5jj5j"] Sep 09 17:37:32.345989 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:32.345957 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9765c4b-2903-40b3-8f60-3b6621be1068" path="/var/lib/kubelet/pods/b9765c4b-2903-40b3-8f60-3b6621be1068/volumes" Sep 09 17:37:32.346361 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:32.346252 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6e65988-3098-4628-85b1-ef4c05caaf6f" path="/var/lib/kubelet/pods/e6e65988-3098-4628-85b1-ef4c05caaf6f/volumes" Sep 09 17:37:35.698953 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:35.698906 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 17:37:38.198747 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:38.198702 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 09 17:37:40.440529 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.440498 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s"] Sep 09 17:37:40.440902 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.440822 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e65988-3098-4628-85b1-ef4c05caaf6f" containerName="redis" Sep 09 17:37:40.440902 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.440836 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e65988-3098-4628-85b1-ef4c05caaf6f" containerName="redis" Sep 09 17:37:40.440902 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.440848 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9616ac32-7a05-4214-a874-262a0d7d137a" containerName="manager" Sep 09 17:37:40.440902 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.440856 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9616ac32-7a05-4214-a874-262a0d7d137a" containerName="manager" Sep 09 17:37:40.440902 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.440866 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9765c4b-2903-40b3-8f60-3b6621be1068" containerName="postgres" Sep 09 17:37:40.440902 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.440876 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9765c4b-2903-40b3-8f60-3b6621be1068" containerName="postgres" Sep 09 17:37:40.441074 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.440930 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9765c4b-2903-40b3-8f60-3b6621be1068" containerName="postgres" Sep 09 17:37:40.441074 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.440938 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9616ac32-7a05-4214-a874-262a0d7d137a" containerName="manager" Sep 09 17:37:40.441074 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.440947 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e65988-3098-4628-85b1-ef4c05caaf6f" containerName="redis" Sep 09 17:37:40.447929 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.447911 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" Sep 09 17:37:40.452238 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.452217 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 09 17:37:40.453469 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.453453 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-4pk25\"" Sep 09 17:37:40.453572 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.453475 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 09 17:37:40.462294 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.462275 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s"] Sep 09 17:37:40.533417 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.533391 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crl5g\" (UniqueName: \"kubernetes.io/projected/7afd2753-0fa9-438f-a6dc-b6be29e0ac2d-kube-api-access-crl5g\") pod \"feast-operator-controller-manager-689b877b4b-95w6s\" (UID: \"7afd2753-0fa9-438f-a6dc-b6be29e0ac2d\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" Sep 09 17:37:40.634863 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.634839 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crl5g\" (UniqueName: \"kubernetes.io/projected/7afd2753-0fa9-438f-a6dc-b6be29e0ac2d-kube-api-access-crl5g\") pod \"feast-operator-controller-manager-689b877b4b-95w6s\" (UID: \"7afd2753-0fa9-438f-a6dc-b6be29e0ac2d\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" Sep 09 17:37:40.644277 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.644242 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crl5g\" (UniqueName: \"kubernetes.io/projected/7afd2753-0fa9-438f-a6dc-b6be29e0ac2d-kube-api-access-crl5g\") pod \"feast-operator-controller-manager-689b877b4b-95w6s\" (UID: \"7afd2753-0fa9-438f-a6dc-b6be29e0ac2d\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" Sep 09 17:37:40.758035 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:40.757956 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" Sep 09 17:37:41.033131 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:41.032638 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s"] Sep 09 17:37:41.038090 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:41.038070 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 17:37:41.921242 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:41.921209 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" event={"ID":"7afd2753-0fa9-438f-a6dc-b6be29e0ac2d","Type":"ContainerStarted","Data":"ce77da7ada73731f5bf68efdfa49c28a078869d3008008b1e94ab0670148ca3d"} Sep 09 17:37:45.699126 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:45.699087 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 17:37:45.939848 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:45.939808 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" event={"ID":"7afd2753-0fa9-438f-a6dc-b6be29e0ac2d","Type":"ContainerStarted","Data":"3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c"} Sep 09 17:37:45.939990 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:45.939916 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" Sep 09 17:37:45.956742 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:45.956474 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" podStartSLOduration=1.737199329 podStartE2EDuration="5.956460249s" totalImagesPullingTime="4.21926092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:37:45.956102167 +0000 UTC m=+1140.196008533" watchObservedRunningTime="2026-09-09 17:37:45.956460249 +0000 UTC m=+1140.196366602" Sep 09 17:37:48.198436 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:48.198397 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 09 17:37:55.698992 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:55.698953 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 17:37:55.699439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:55.699104 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:37:56.947744 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:56.947718 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" Sep 09 17:37:58.198718 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:58.198675 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 09 17:37:58.199151 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:58.198796 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:37:59.793941 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.793919 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:37:59.817076 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.817052 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:37:59.879181 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.879154 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-feast-data\") pod \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " Sep 09 17:37:59.879332 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.879202 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-online-tls\") pod \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " Sep 09 17:37:59.879332 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.879255 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhjbt\" (UniqueName: \"kubernetes.io/projected/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-kube-api-access-xhjbt\") pod \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " Sep 09 17:37:59.879332 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.879277 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-registry-tls\") pod \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " Sep 09 17:37:59.879332 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.879307 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7d486c5c-b5b8-4868-99c2-537b9b8921b1-feast-data\") pod \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " Sep 09 17:37:59.879539 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.879432 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh24q\" (UniqueName: \"kubernetes.io/projected/7d486c5c-b5b8-4868-99c2-537b9b8921b1-kube-api-access-xh24q\") pod \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\" (UID: \"7d486c5c-b5b8-4868-99c2-537b9b8921b1\") " Sep 09 17:37:59.879539 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.879477 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-registry-tls\") pod \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " Sep 09 17:37:59.879539 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.879516 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-online-tls\") pod \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\" (UID: \"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18\") " Sep 09 17:37:59.879859 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.879832 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d486c5c-b5b8-4868-99c2-537b9b8921b1-feast-data" (OuterVolumeSpecName: "feast-data") pod "7d486c5c-b5b8-4868-99c2-537b9b8921b1" (UID: "7d486c5c-b5b8-4868-99c2-537b9b8921b1"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:37:59.881823 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.881779 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" (UID: "9b59d9a4-6405-4b01-bbdc-5dfa67bfff18"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:37:59.881971 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.881863 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-online-tls" (OuterVolumeSpecName: "online-tls") pod "7d486c5c-b5b8-4868-99c2-537b9b8921b1" (UID: "7d486c5c-b5b8-4868-99c2-537b9b8921b1"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:37:59.881971 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.881924 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d486c5c-b5b8-4868-99c2-537b9b8921b1-kube-api-access-xh24q" (OuterVolumeSpecName: "kube-api-access-xh24q") pod "7d486c5c-b5b8-4868-99c2-537b9b8921b1" (UID: "7d486c5c-b5b8-4868-99c2-537b9b8921b1"). InnerVolumeSpecName "kube-api-access-xh24q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:37:59.882080 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.882034 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-kube-api-access-xhjbt" (OuterVolumeSpecName: "kube-api-access-xhjbt") pod "9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" (UID: "9b59d9a4-6405-4b01-bbdc-5dfa67bfff18"). InnerVolumeSpecName "kube-api-access-xhjbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:37:59.882152 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.882127 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-online-tls" (OuterVolumeSpecName: "online-tls") pod "9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" (UID: "9b59d9a4-6405-4b01-bbdc-5dfa67bfff18"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:37:59.882207 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.882127 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7d486c5c-b5b8-4868-99c2-537b9b8921b1" (UID: "7d486c5c-b5b8-4868-99c2-537b9b8921b1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:37:59.889207 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.889180 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-feast-data" (OuterVolumeSpecName: "feast-data") pod "9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" (UID: "9b59d9a4-6405-4b01-bbdc-5dfa67bfff18"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:37:59.980796 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.980746 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhjbt\" (UniqueName: \"kubernetes.io/projected/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-kube-api-access-xhjbt\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:59.980796 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.980766 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:59.980796 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.980775 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7d486c5c-b5b8-4868-99c2-537b9b8921b1-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:59.980796 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.980784 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh24q\" (UniqueName: \"kubernetes.io/projected/7d486c5c-b5b8-4868-99c2-537b9b8921b1-kube-api-access-xh24q\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:59.980796 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.980791 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:59.980796 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.980799 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:59.981026 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.980806 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:59.981026 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.980814 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7d486c5c-b5b8-4868-99c2-537b9b8921b1-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:37:59.995022 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.994994 2553 generic.go:356] "Generic (PLEG): container finished" podID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerID="5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c" exitCode=137 Sep 09 17:37:59.995111 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.995058 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" Sep 09 17:37:59.995111 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.995071 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" event={"ID":"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18","Type":"ContainerDied","Data":"5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c"} Sep 09 17:37:59.995192 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.995145 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr" event={"ID":"9b59d9a4-6405-4b01-bbdc-5dfa67bfff18","Type":"ContainerDied","Data":"80249c5caab7c0b99cc4a7e37e6917bf7855fb729ece68bdedd3c0a3c3c0373b"} Sep 09 17:37:59.995192 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.995177 2553 scope.go:117] "RemoveContainer" containerID="44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747" Sep 09 17:37:59.997392 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.997369 2553 generic.go:356] "Generic (PLEG): container finished" podID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerID="c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592" exitCode=137 Sep 09 17:37:59.997472 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.997420 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" event={"ID":"7d486c5c-b5b8-4868-99c2-537b9b8921b1","Type":"ContainerDied","Data":"c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592"} Sep 09 17:37:59.997472 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.997439 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" Sep 09 17:37:59.997472 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:37:59.997459 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65" event={"ID":"7d486c5c-b5b8-4868-99c2-537b9b8921b1","Type":"ContainerDied","Data":"f3e0c3101e443bef2668529a18a68da216242670f8d2b2ed430a2c15b9fc39bc"} Sep 09 17:38:00.004347 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.004331 2553 scope.go:117] "RemoveContainer" containerID="5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c" Sep 09 17:38:00.015805 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.015790 2553 scope.go:117] "RemoveContainer" containerID="0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2" Sep 09 17:38:00.023059 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.023039 2553 scope.go:117] "RemoveContainer" containerID="cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a" Sep 09 17:38:00.024275 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.024253 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr"] Sep 09 17:38:00.026054 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.026035 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b5d96f645-lpvjr"] Sep 09 17:38:00.030359 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.030343 2553 scope.go:117] "RemoveContainer" containerID="44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747" Sep 09 17:38:00.030601 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.030585 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747\": container with ID starting with 44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747 not found: ID does not exist" containerID="44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747" Sep 09 17:38:00.030715 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.030606 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747"} err="failed to get container status \"44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747\": rpc error: code = NotFound desc = could not find container \"44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747\": container with ID starting with 44ab3b7f95c6e694a4fa6c41936a907b3cbb050c7c82489c8159fa1d81fa2747 not found: ID does not exist" Sep 09 17:38:00.030715 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.030636 2553 scope.go:117] "RemoveContainer" containerID="5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c" Sep 09 17:38:00.030894 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.030880 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c\": container with ID starting with 5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c not found: ID does not exist" containerID="5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c" Sep 09 17:38:00.030931 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.030897 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c"} err="failed to get container status \"5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c\": rpc error: code = NotFound desc = could not find container \"5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c\": container with ID starting with 5f0c19a7a7c526483b001f927218b07f84bd86c3337d011b1a80f3247165197c not found: ID does not exist" Sep 09 17:38:00.030931 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.030909 2553 scope.go:117] "RemoveContainer" containerID="0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2" Sep 09 17:38:00.031124 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.031108 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2\": container with ID starting with 0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2 not found: ID does not exist" containerID="0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2" Sep 09 17:38:00.031158 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.031128 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2"} err="failed to get container status \"0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2\": rpc error: code = NotFound desc = could not find container \"0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2\": container with ID starting with 0bd8d2cfa68a62ad5db045fa56195eeb59738f3be41d09872c60fd15181730e2 not found: ID does not exist" Sep 09 17:38:00.031158 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.031140 2553 scope.go:117] "RemoveContainer" containerID="cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a" Sep 09 17:38:00.031354 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.031336 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a\": container with ID starting with cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a not found: ID does not exist" containerID="cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a" Sep 09 17:38:00.031388 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.031361 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a"} err="failed to get container status \"cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a\": rpc error: code = NotFound desc = could not find container \"cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a\": container with ID starting with cd1ae33dc7986c430ed5d3ab3c93c9078f5bf4e688984c86ba661a4aca90157a not found: ID does not exist" Sep 09 17:38:00.031388 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.031378 2553 scope.go:117] "RemoveContainer" containerID="feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266" Sep 09 17:38:00.037082 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.037059 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65"] Sep 09 17:38:00.038859 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.038846 2553 scope.go:117] "RemoveContainer" containerID="c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592" Sep 09 17:38:00.042378 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.042356 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-66c846fd6-5bm65"] Sep 09 17:38:00.046333 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.046318 2553 scope.go:117] "RemoveContainer" containerID="9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a" Sep 09 17:38:00.052777 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.052761 2553 scope.go:117] "RemoveContainer" containerID="1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d" Sep 09 17:38:00.059449 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.059434 2553 scope.go:117] "RemoveContainer" containerID="feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266" Sep 09 17:38:00.059705 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.059689 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266\": container with ID starting with feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266 not found: ID does not exist" containerID="feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266" Sep 09 17:38:00.059758 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.059710 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266"} err="failed to get container status \"feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266\": rpc error: code = NotFound desc = could not find container \"feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266\": container with ID starting with feb1706c1120212139bb7904018ec36fba52b5418a76e0b4e50c69af82056266 not found: ID does not exist" Sep 09 17:38:00.059758 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.059725 2553 scope.go:117] "RemoveContainer" containerID="c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592" Sep 09 17:38:00.059949 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.059932 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592\": container with ID starting with c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592 not found: ID does not exist" containerID="c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592" Sep 09 17:38:00.060006 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.059958 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592"} err="failed to get container status \"c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592\": rpc error: code = NotFound desc = could not find container \"c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592\": container with ID starting with c02ef996c4ba9cb7ff31d5f55473b502441e8621ac3b84892116bc4d37a20592 not found: ID does not exist" Sep 09 17:38:00.060006 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.059981 2553 scope.go:117] "RemoveContainer" containerID="9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a" Sep 09 17:38:00.060261 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.060243 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a\": container with ID starting with 9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a not found: ID does not exist" containerID="9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a" Sep 09 17:38:00.060302 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.060269 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a"} err="failed to get container status \"9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a\": rpc error: code = NotFound desc = could not find container \"9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a\": container with ID starting with 9860e6b59e75392f4a4938bfabc02b2a0affacc36b6aab2bbcdccac94a54e94a not found: ID does not exist" Sep 09 17:38:00.060302 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.060286 2553 scope.go:117] "RemoveContainer" containerID="1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d" Sep 09 17:38:00.060511 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.060494 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d\": container with ID starting with 1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d not found: ID does not exist" containerID="1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d" Sep 09 17:38:00.060563 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.060517 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d"} err="failed to get container status \"1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d\": rpc error: code = NotFound desc = could not find container \"1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d\": container with ID starting with 1cf33727a5b5fa666b3f87c0d1b7fb87b60540e8ee7cacde4ac9a66fcd98274d not found: ID does not exist" Sep 09 17:38:00.346838 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.346767 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" path="/var/lib/kubelet/pods/7d486c5c-b5b8-4868-99c2-537b9b8921b1/volumes" Sep 09 17:38:00.347271 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.347258 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" path="/var/lib/kubelet/pods/9b59d9a4-6405-4b01-bbdc-5dfa67bfff18/volumes" Sep 09 17:38:00.371901 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.371873 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89"] Sep 09 17:38:00.372145 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372134 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="feast-init" Sep 09 17:38:00.372189 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372146 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="feast-init" Sep 09 17:38:00.372189 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372159 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="feast-init" Sep 09 17:38:00.372189 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372164 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="feast-init" Sep 09 17:38:00.372189 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372174 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="feast-apply" Sep 09 17:38:00.372189 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372180 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="feast-apply" Sep 09 17:38:00.372189 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372185 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="feast-apply" Sep 09 17:38:00.372189 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372190 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="feast-apply" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372200 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="online" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372204 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="online" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372210 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="online" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372214 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="online" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372222 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="registry" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372227 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="registry" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372235 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="registry" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372240 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="registry" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372282 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="online" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372290 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="registry" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372298 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b59d9a4-6405-4b01-bbdc-5dfa67bfff18" containerName="online" Sep 09 17:38:00.372402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.372305 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d486c5c-b5b8-4868-99c2-537b9b8921b1" containerName="registry" Sep 09 17:38:00.376689 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.376674 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.379656 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.379632 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 17:38:00.379656 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.379647 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 17:38:00.379847 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.379697 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 17:38:00.379847 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.379652 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 17:38:00.379847 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.379736 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 17:38:00.380005 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.379898 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 17:38:00.380005 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.379980 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-qqnkz\"" Sep 09 17:38:00.386451 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.386429 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89"] Sep 09 17:38:00.484438 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.484411 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.484548 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.484450 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d6f814-91e6-4a7d-803f-0d65782e475b-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.484548 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.484472 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrs6k\" (UniqueName: \"kubernetes.io/projected/e3d6f814-91e6-4a7d-803f-0d65782e475b-kube-api-access-qrs6k\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.484548 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.484492 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.484548 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.484519 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.484702 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.484564 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.585430 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.585404 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d6f814-91e6-4a7d-803f-0d65782e475b-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.585558 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.585438 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrs6k\" (UniqueName: \"kubernetes.io/projected/e3d6f814-91e6-4a7d-803f-0d65782e475b-kube-api-access-qrs6k\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.585558 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.585455 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.585558 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.585479 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.585558 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.585498 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.585558 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.585524 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.585558 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.585549 2553 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 17:38:00.585895 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.585602 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-ui-tls podName:e3d6f814-91e6-4a7d-803f-0d65782e475b nodeName:}" failed. No retries permitted until 2026-09-09 17:38:01.085586868 +0000 UTC m=+1155.325493199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-xbp89" (UID: "e3d6f814-91e6-4a7d-803f-0d65782e475b") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 17:38:00.585895 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.585723 2553 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 17:38:00.585895 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:38:00.585790 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-offline-tls podName:e3d6f814-91e6-4a7d-803f-0d65782e475b nodeName:}" failed. No retries permitted until 2026-09-09 17:38:01.085771406 +0000 UTC m=+1155.325677754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-xbp89" (UID: "e3d6f814-91e6-4a7d-803f-0d65782e475b") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 17:38:00.586003 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.585930 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d6f814-91e6-4a7d-803f-0d65782e475b-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.587941 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.587919 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.588032 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.587977 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:00.593290 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:00.593259 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrs6k\" (UniqueName: \"kubernetes.io/projected/e3d6f814-91e6-4a7d-803f-0d65782e475b-kube-api-access-qrs6k\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:01.089730 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:01.089695 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:01.090093 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:01.089745 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:01.092135 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:01.092102 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:01.092135 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:01.092128 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xbp89\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:01.291638 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:01.291584 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:01.568285 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:01.568257 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89"] Sep 09 17:38:01.569419 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:38:01.569395 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3d6f814_91e6_4a7d_803f_0d65782e475b.slice/crio-7edb98f61495d31bf1af85434e24f06c19e4479f743b7c30befa72709a5ca2b1 WatchSource:0}: Error finding container 7edb98f61495d31bf1af85434e24f06c19e4479f743b7c30befa72709a5ca2b1: Status 404 returned error can't find the container with id 7edb98f61495d31bf1af85434e24f06c19e4479f743b7c30befa72709a5ca2b1 Sep 09 17:38:02.008289 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:02.008252 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerStarted","Data":"7edb98f61495d31bf1af85434e24f06c19e4479f743b7c30befa72709a5ca2b1"} Sep 09 17:38:17.080546 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:17.080509 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerStarted","Data":"2551be4d95ca9a0f6a92c9ef302263eed6516cf4f0fa573e53b788eec28a3fa0"} Sep 09 17:38:21.097603 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:21.097570 2553 generic.go:356] "Generic (PLEG): container finished" podID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerID="2551be4d95ca9a0f6a92c9ef302263eed6516cf4f0fa573e53b788eec28a3fa0" exitCode=0 Sep 09 17:38:21.098020 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:21.097644 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerDied","Data":"2551be4d95ca9a0f6a92c9ef302263eed6516cf4f0fa573e53b788eec28a3fa0"} Sep 09 17:38:22.105341 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:22.105307 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerStarted","Data":"4ad20da785189140ecf720990611226941b719742569d26ddac45afd3b03557d"} Sep 09 17:38:27.126019 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:27.125940 2553 generic.go:356] "Generic (PLEG): container finished" podID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerID="4ad20da785189140ecf720990611226941b719742569d26ddac45afd3b03557d" exitCode=0 Sep 09 17:38:27.126019 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:27.126009 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerDied","Data":"4ad20da785189140ecf720990611226941b719742569d26ddac45afd3b03557d"} Sep 09 17:38:28.136059 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.136018 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerStarted","Data":"78f99edb1f553fb6e696750619e25039f647500b900f7ac4739520ffd06f5ca0"} Sep 09 17:38:28.136545 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.136084 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerStarted","Data":"b6985ea895ecbed097836e4fa31a06367225fc8255afb6b3d2e6ea16bf24485c"} Sep 09 17:38:28.136545 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.136106 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerStarted","Data":"bcf011621bf0ddabf1a8a94eaeaa377f6d5b6054bd1511436b0940e0ebd1b434"} Sep 09 17:38:28.136545 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.136125 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerStarted","Data":"e39f63d0e18e5748c49ce51f6f9b495ec6de739d27c297a514a3e8efe8e22eb1"} Sep 09 17:38:28.160401 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.159591 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podStartSLOduration=13.626309053 podStartE2EDuration="28.159563169s" totalImagesPullingTime="14.533254116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:38:28.157931316 +0000 UTC m=+1182.397837685" watchObservedRunningTime="2026-09-09 17:38:28.159563169 +0000 UTC m=+1182.399469522" Sep 09 17:38:28.292440 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.292398 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:28.292654 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.292462 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:28.292654 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.292486 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:28.292654 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.292505 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:28.294784 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.294750 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 09 17:38:28.295012 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.294772 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Sep 09 17:38:28.295205 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.294815 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused" Sep 09 17:38:28.295315 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:28.294811 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 09 17:38:31.292329 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:31.292287 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:31.292914 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:31.292873 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused" Sep 09 17:38:31.293042 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:31.292707 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 09 17:38:31.293042 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:31.292776 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 09 17:38:31.293042 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:31.292887 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:31.293216 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:31.292646 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Sep 09 17:38:31.293369 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:31.293304 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:31.293369 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:31.293343 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:34.292822 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:34.292638 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 09 17:38:34.292822 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:34.292675 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Sep 09 17:38:34.292822 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:34.292699 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused" Sep 09 17:38:34.292822 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:34.292787 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 09 17:38:37.293492 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:37.293461 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:37.293940 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:37.293612 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:37.293940 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:37.293810 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:37.294368 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:37.294345 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:37.294460 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:37.294411 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:37.294684 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:37.294667 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:37.298807 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:37.298789 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:38:38.185792 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:38:38.185764 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:39:39.935890 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:39.935806 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89"] Sep 09 17:39:39.936430 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:39.936096 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="registry" containerID="cri-o://e39f63d0e18e5748c49ce51f6f9b495ec6de739d27c297a514a3e8efe8e22eb1" gracePeriod=30 Sep 09 17:39:39.936430 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:39.936174 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="offline" containerID="cri-o://b6985ea895ecbed097836e4fa31a06367225fc8255afb6b3d2e6ea16bf24485c" gracePeriod=30 Sep 09 17:39:39.936430 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:39.936202 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="ui" containerID="cri-o://78f99edb1f553fb6e696750619e25039f647500b900f7ac4739520ffd06f5ca0" gracePeriod=30 Sep 09 17:39:39.936430 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:39.936181 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="online" containerID="cri-o://bcf011621bf0ddabf1a8a94eaeaa377f6d5b6054bd1511436b0940e0ebd1b434" gracePeriod=30 Sep 09 17:39:40.377993 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.377904 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb"] Sep 09 17:39:40.382405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.382384 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.385084 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.385057 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-4pxt4\"" Sep 09 17:39:40.394002 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.393972 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb"] Sep 09 17:39:40.465418 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.465382 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.465731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.465700 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.465731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.465730 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9642b6-12a2-4a9d-943a-c00ddb31583f-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.465905 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.465761 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.465905 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.465835 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.466001 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.465903 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m7sh\" (UniqueName: \"kubernetes.io/projected/dd9642b6-12a2-4a9d-943a-c00ddb31583f-kube-api-access-2m7sh\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.566719 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.566687 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m7sh\" (UniqueName: \"kubernetes.io/projected/dd9642b6-12a2-4a9d-943a-c00ddb31583f-kube-api-access-2m7sh\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.566817 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.566749 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.566817 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.566785 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.566817 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.566807 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9642b6-12a2-4a9d-943a-c00ddb31583f-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.566934 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.566851 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.566934 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.566884 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.567361 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.567316 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9642b6-12a2-4a9d-943a-c00ddb31583f-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.569379 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.569356 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.569479 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.569362 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.569479 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.569362 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.569846 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.569824 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.574197 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.574171 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m7sh\" (UniqueName: \"kubernetes.io/projected/dd9642b6-12a2-4a9d-943a-c00ddb31583f-kube-api-access-2m7sh\") pod \"feast-simple-feast-setup-57576b8fcf-xqqgb\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.693313 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.693280 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:40.973837 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:40.973800 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb"] Sep 09 17:39:40.975694 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:39:40.975670 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd9642b6_12a2_4a9d_943a_c00ddb31583f.slice/crio-ec18287c98570c5266ad6932981c69f6d8fa2e37548c8df51bcce6736133d1a4 WatchSource:0}: Error finding container ec18287c98570c5266ad6932981c69f6d8fa2e37548c8df51bcce6736133d1a4: Status 404 returned error can't find the container with id ec18287c98570c5266ad6932981c69f6d8fa2e37548c8df51bcce6736133d1a4 Sep 09 17:39:41.414253 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:41.414225 2553 generic.go:356] "Generic (PLEG): container finished" podID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerID="78f99edb1f553fb6e696750619e25039f647500b900f7ac4739520ffd06f5ca0" exitCode=0 Sep 09 17:39:41.414253 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:41.414252 2553 generic.go:356] "Generic (PLEG): container finished" podID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerID="bcf011621bf0ddabf1a8a94eaeaa377f6d5b6054bd1511436b0940e0ebd1b434" exitCode=0 Sep 09 17:39:41.414504 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:41.414296 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerDied","Data":"78f99edb1f553fb6e696750619e25039f647500b900f7ac4739520ffd06f5ca0"} Sep 09 17:39:41.414504 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:41.414334 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerDied","Data":"bcf011621bf0ddabf1a8a94eaeaa377f6d5b6054bd1511436b0940e0ebd1b434"} Sep 09 17:39:41.415773 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:41.415748 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerStarted","Data":"159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3"} Sep 09 17:39:41.415885 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:41.415781 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerStarted","Data":"ec18287c98570c5266ad6932981c69f6d8fa2e37548c8df51bcce6736133d1a4"} Sep 09 17:39:46.435186 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:46.435155 2553 generic.go:356] "Generic (PLEG): container finished" podID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerID="159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3" exitCode=0 Sep 09 17:39:46.435555 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:46.435233 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerDied","Data":"159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3"} Sep 09 17:39:47.294562 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:47.294517 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Sep 09 17:39:47.441199 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:47.441173 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerStarted","Data":"adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62"} Sep 09 17:39:48.183400 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:48.183360 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 09 17:39:52.461271 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:52.461199 2553 generic.go:356] "Generic (PLEG): container finished" podID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerID="adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62" exitCode=0 Sep 09 17:39:52.461271 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:52.461250 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerDied","Data":"adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62"} Sep 09 17:39:53.473427 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:53.473388 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerStarted","Data":"83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e"} Sep 09 17:39:53.474015 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:53.473449 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerStarted","Data":"a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22"} Sep 09 17:39:53.474015 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:53.473470 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerStarted","Data":"02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8"} Sep 09 17:39:53.474015 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:53.473488 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerStarted","Data":"582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38"} Sep 09 17:39:53.498831 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:53.498772 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podStartSLOduration=13.49875056 podStartE2EDuration="13.49875056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:53.497365049 +0000 UTC m=+1267.737271402" watchObservedRunningTime="2026-09-09 17:39:53.49875056 +0000 UTC m=+1267.738656914" Sep 09 17:39:55.694533 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:55.694498 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:55.695114 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:55.695083 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:55.695114 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:55.695119 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:55.695305 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:55.695138 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:39:55.696477 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:55.696443 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 09 17:39:55.696662 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:55.696482 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Sep 09 17:39:55.696776 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:55.696478 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 17:39:55.696776 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:55.696536 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 09 17:39:57.295052 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:57.295003 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Sep 09 17:39:58.183138 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:58.183089 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 09 17:39:58.694670 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:58.694606 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 09 17:39:58.695255 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:58.694611 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Sep 09 17:39:58.695386 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:58.694611 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 09 17:39:58.695386 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:39:58.694666 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 17:40:00.694513 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:00.694465 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:00.695000 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:00.694527 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:00.695000 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:00.694548 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:00.695000 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:00.694566 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:01.695339 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:01.695308 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:01.695819 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:01.695402 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:01.695819 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:01.695681 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:01.696228 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:01.696206 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:01.696333 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:01.696262 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:01.696451 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:01.696437 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:01.700608 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:01.700591 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:02.524021 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:02.523992 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:40:07.294768 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:07.294717 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Sep 09 17:40:07.295225 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:07.294900 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:40:08.183641 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:08.183591 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 09 17:40:08.183918 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:08.183768 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:40:10.554495 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.554464 2553 generic.go:356] "Generic (PLEG): container finished" podID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerID="b6985ea895ecbed097836e4fa31a06367225fc8255afb6b3d2e6ea16bf24485c" exitCode=137 Sep 09 17:40:10.554836 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.554497 2553 generic.go:356] "Generic (PLEG): container finished" podID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerID="e39f63d0e18e5748c49ce51f6f9b495ec6de739d27c297a514a3e8efe8e22eb1" exitCode=137 Sep 09 17:40:10.554836 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.554528 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerDied","Data":"b6985ea895ecbed097836e4fa31a06367225fc8255afb6b3d2e6ea16bf24485c"} Sep 09 17:40:10.554836 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.554570 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerDied","Data":"e39f63d0e18e5748c49ce51f6f9b495ec6de739d27c297a514a3e8efe8e22eb1"} Sep 09 17:40:10.571857 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.571838 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:40:10.722662 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.722552 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-offline-tls\") pod \"e3d6f814-91e6-4a7d-803f-0d65782e475b\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " Sep 09 17:40:10.722662 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.722590 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-ui-tls\") pod \"e3d6f814-91e6-4a7d-803f-0d65782e475b\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " Sep 09 17:40:10.722662 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.722614 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-online-tls\") pod \"e3d6f814-91e6-4a7d-803f-0d65782e475b\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " Sep 09 17:40:10.722920 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.722703 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-registry-tls\") pod \"e3d6f814-91e6-4a7d-803f-0d65782e475b\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " Sep 09 17:40:10.722920 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.722735 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d6f814-91e6-4a7d-803f-0d65782e475b-feast-data\") pod \"e3d6f814-91e6-4a7d-803f-0d65782e475b\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " Sep 09 17:40:10.722920 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.722757 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrs6k\" (UniqueName: \"kubernetes.io/projected/e3d6f814-91e6-4a7d-803f-0d65782e475b-kube-api-access-qrs6k\") pod \"e3d6f814-91e6-4a7d-803f-0d65782e475b\" (UID: \"e3d6f814-91e6-4a7d-803f-0d65782e475b\") " Sep 09 17:40:10.723339 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.723309 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d6f814-91e6-4a7d-803f-0d65782e475b-feast-data" (OuterVolumeSpecName: "feast-data") pod "e3d6f814-91e6-4a7d-803f-0d65782e475b" (UID: "e3d6f814-91e6-4a7d-803f-0d65782e475b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:40:10.724893 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.724868 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-online-tls" (OuterVolumeSpecName: "online-tls") pod "e3d6f814-91e6-4a7d-803f-0d65782e475b" (UID: "e3d6f814-91e6-4a7d-803f-0d65782e475b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:40:10.725202 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.725179 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e3d6f814-91e6-4a7d-803f-0d65782e475b" (UID: "e3d6f814-91e6-4a7d-803f-0d65782e475b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:40:10.725276 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.725200 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e3d6f814-91e6-4a7d-803f-0d65782e475b" (UID: "e3d6f814-91e6-4a7d-803f-0d65782e475b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:40:10.725276 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.725250 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e3d6f814-91e6-4a7d-803f-0d65782e475b" (UID: "e3d6f814-91e6-4a7d-803f-0d65782e475b"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:40:10.725276 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.725266 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d6f814-91e6-4a7d-803f-0d65782e475b-kube-api-access-qrs6k" (OuterVolumeSpecName: "kube-api-access-qrs6k") pod "e3d6f814-91e6-4a7d-803f-0d65782e475b" (UID: "e3d6f814-91e6-4a7d-803f-0d65782e475b"). InnerVolumeSpecName "kube-api-access-qrs6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:40:10.823467 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.823440 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:40:10.823467 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.823465 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d6f814-91e6-4a7d-803f-0d65782e475b-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:40:10.823650 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.823477 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrs6k\" (UniqueName: \"kubernetes.io/projected/e3d6f814-91e6-4a7d-803f-0d65782e475b-kube-api-access-qrs6k\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:40:10.823650 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.823489 2553 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-offline-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:40:10.823650 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.823501 2553 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-ui-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:40:10.823650 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:10.823511 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e3d6f814-91e6-4a7d-803f-0d65782e475b-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:40:11.560785 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:11.560755 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" event={"ID":"e3d6f814-91e6-4a7d-803f-0d65782e475b","Type":"ContainerDied","Data":"7edb98f61495d31bf1af85434e24f06c19e4479f743b7c30befa72709a5ca2b1"} Sep 09 17:40:11.561156 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:11.560797 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89" Sep 09 17:40:11.561156 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:11.560801 2553 scope.go:117] "RemoveContainer" containerID="78f99edb1f553fb6e696750619e25039f647500b900f7ac4739520ffd06f5ca0" Sep 09 17:40:11.569993 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:11.569975 2553 scope.go:117] "RemoveContainer" containerID="b6985ea895ecbed097836e4fa31a06367225fc8255afb6b3d2e6ea16bf24485c" Sep 09 17:40:11.577318 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:11.577301 2553 scope.go:117] "RemoveContainer" containerID="bcf011621bf0ddabf1a8a94eaeaa377f6d5b6054bd1511436b0940e0ebd1b434" Sep 09 17:40:11.585402 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:11.585378 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89"] Sep 09 17:40:11.586588 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:11.586557 2553 scope.go:117] "RemoveContainer" containerID="e39f63d0e18e5748c49ce51f6f9b495ec6de739d27c297a514a3e8efe8e22eb1" Sep 09 17:40:11.592428 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:11.592406 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xbp89"] Sep 09 17:40:11.594111 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:11.594097 2553 scope.go:117] "RemoveContainer" containerID="4ad20da785189140ecf720990611226941b719742569d26ddac45afd3b03557d" Sep 09 17:40:11.600721 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:11.600697 2553 scope.go:117] "RemoveContainer" containerID="2551be4d95ca9a0f6a92c9ef302263eed6516cf4f0fa573e53b788eec28a3fa0" Sep 09 17:40:12.346883 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:40:12.346848 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" path="/var/lib/kubelet/pods/e3d6f814-91e6-4a7d-803f-0d65782e475b/volumes" Sep 09 17:41:05.605575 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.605543 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz"] Sep 09 17:41:05.606428 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606400 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="ui" Sep 09 17:41:05.606428 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606428 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="ui" Sep 09 17:41:05.606648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606455 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="feast-apply" Sep 09 17:41:05.606648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606465 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="feast-apply" Sep 09 17:41:05.606648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606485 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="online" Sep 09 17:41:05.606648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606494 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="online" Sep 09 17:41:05.606648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606508 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="registry" Sep 09 17:41:05.606648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606516 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="registry" Sep 09 17:41:05.606648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606533 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="offline" Sep 09 17:41:05.606648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606542 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="offline" Sep 09 17:41:05.606648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606565 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="feast-init" Sep 09 17:41:05.606648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606580 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="feast-init" Sep 09 17:41:05.607286 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606798 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="online" Sep 09 17:41:05.607286 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606814 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="offline" Sep 09 17:41:05.607286 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606830 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="registry" Sep 09 17:41:05.607286 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.606841 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d6f814-91e6-4a7d-803f-0d65782e475b" containerName="ui" Sep 09 17:41:05.613608 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.613583 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.616487 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.616466 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 17:41:05.617847 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.617827 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-75hjg\"" Sep 09 17:41:05.617957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.617853 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 17:41:05.617957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.617877 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 17:41:05.617957 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.617853 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 17:41:05.618207 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.618193 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 17:41:05.618386 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.618370 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 17:41:05.624519 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.624500 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz"] Sep 09 17:41:05.717747 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.717726 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.717856 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.717761 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.717856 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.717784 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7a7eca-177b-4990-b8a0-6034db11ce80-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.717944 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.717856 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0f7a7eca-177b-4990-b8a0-6034db11ce80-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.717944 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.717882 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xk6\" (UniqueName: \"kubernetes.io/projected/0f7a7eca-177b-4990-b8a0-6034db11ce80-kube-api-access-k2xk6\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.717944 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.717915 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.819048 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.819023 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7a7eca-177b-4990-b8a0-6034db11ce80-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.819166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.819057 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0f7a7eca-177b-4990-b8a0-6034db11ce80-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.819166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.819074 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xk6\" (UniqueName: \"kubernetes.io/projected/0f7a7eca-177b-4990-b8a0-6034db11ce80-kube-api-access-k2xk6\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.819166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.819091 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.819166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.819133 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.819166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.819160 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.819413 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:41:05.819344 2553 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 17:41:05.819453 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:41:05.819426 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-online-tls podName:0f7a7eca-177b-4990-b8a0-6034db11ce80 nodeName:}" failed. No retries permitted until 2026-09-09 17:41:06.319403445 +0000 UTC m=+1340.559309787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-online-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-94qhz" (UID: "0f7a7eca-177b-4990-b8a0-6034db11ce80") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 17:41:05.819506 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.819462 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7a7eca-177b-4990-b8a0-6034db11ce80-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.819812 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.819790 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0f7a7eca-177b-4990-b8a0-6034db11ce80-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.821520 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.821494 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.821640 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.821597 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:05.828817 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:05.828787 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xk6\" (UniqueName: \"kubernetes.io/projected/0f7a7eca-177b-4990-b8a0-6034db11ce80-kube-api-access-k2xk6\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:06.323549 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:06.323516 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:06.325885 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:06.325862 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-94qhz\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:06.522355 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:06.522325 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:06.796871 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:06.796850 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz"] Sep 09 17:41:06.798613 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:41:06.798587 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f7a7eca_177b_4990_b8a0_6034db11ce80.slice/crio-a00fc82e36e5af711b72a9f90f221019bb0592e0188a258aa8af0256c729863f WatchSource:0}: Error finding container a00fc82e36e5af711b72a9f90f221019bb0592e0188a258aa8af0256c729863f: Status 404 returned error can't find the container with id a00fc82e36e5af711b72a9f90f221019bb0592e0188a258aa8af0256c729863f Sep 09 17:41:07.768413 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:07.768376 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerStarted","Data":"06899502ca059f666b3e6a67fa7b248367212d80d36318fe3c057a8a64c91049"} Sep 09 17:41:07.768565 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:07.768429 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerStarted","Data":"a00fc82e36e5af711b72a9f90f221019bb0592e0188a258aa8af0256c729863f"} Sep 09 17:41:11.784738 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:11.784703 2553 generic.go:356] "Generic (PLEG): container finished" podID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerID="06899502ca059f666b3e6a67fa7b248367212d80d36318fe3c057a8a64c91049" exitCode=0 Sep 09 17:41:11.785084 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:11.784774 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerDied","Data":"06899502ca059f666b3e6a67fa7b248367212d80d36318fe3c057a8a64c91049"} Sep 09 17:41:12.790880 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:12.790847 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerStarted","Data":"3584db2e2b90e78538b8ab15cc2dbfd9a39dc2b8b20a6af74e8f563bfe35c0ab"} Sep 09 17:41:35.879583 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:35.879547 2553 generic.go:356] "Generic (PLEG): container finished" podID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerID="3584db2e2b90e78538b8ab15cc2dbfd9a39dc2b8b20a6af74e8f563bfe35c0ab" exitCode=0 Sep 09 17:41:35.879999 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:35.879637 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerDied","Data":"3584db2e2b90e78538b8ab15cc2dbfd9a39dc2b8b20a6af74e8f563bfe35c0ab"} Sep 09 17:41:36.897700 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:36.897660 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerStarted","Data":"e98b5a0f5b615e6a2d7f7db5b2dad056f399f4d8a325cd29af6cf892cdf1508a"} Sep 09 17:41:36.898120 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:36.897728 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerStarted","Data":"cd89d259001d74c18b96210cb1cb49c36fc1272ae55a5fc2a2f690bd0710151a"} Sep 09 17:41:36.898120 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:36.897749 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerStarted","Data":"833bc352856e10c104f2ebbc35f71a6c7ec8e03b578a300b24f67491ad8b00c1"} Sep 09 17:41:36.922708 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:36.922656 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podStartSLOduration=31.922613545 podStartE2EDuration="31.922613545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:41:36.920906189 +0000 UTC m=+1371.160812543" watchObservedRunningTime="2026-09-09 17:41:36.922613545 +0000 UTC m=+1371.162519902" Sep 09 17:41:39.522779 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:39.522749 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:39.523245 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:39.522834 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:39.523245 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:39.522856 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:39.524926 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:39.524888 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Sep 09 17:41:39.525063 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:39.524899 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused" Sep 09 17:41:39.525063 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:39.524907 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 09 17:41:42.522899 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:42.522850 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Sep 09 17:41:42.522899 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:42.522878 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 09 17:41:42.523457 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:42.522857 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused" Sep 09 17:41:45.523592 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:45.523554 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:45.524392 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:45.524362 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:45.524501 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:45.524418 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:45.524501 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:45.524474 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:45.940093 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:45.940060 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:45.940608 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:45.940591 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:46.522757 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:46.522728 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:46.526755 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:46.526725 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 17:41:49.267833 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:49.267803 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:41:50.014223 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:41:50.014196 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:42:47.527775 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:47.527732 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 17:42:52.192889 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:52.192855 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz"] Sep 09 17:42:52.193457 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:52.193271 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="online" containerID="cri-o://833bc352856e10c104f2ebbc35f71a6c7ec8e03b578a300b24f67491ad8b00c1" gracePeriod=30 Sep 09 17:42:52.193457 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:52.193349 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="ui" containerID="cri-o://e98b5a0f5b615e6a2d7f7db5b2dad056f399f4d8a325cd29af6cf892cdf1508a" gracePeriod=30 Sep 09 17:42:52.193457 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:52.193430 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="offline" containerID="cri-o://cd89d259001d74c18b96210cb1cb49c36fc1272ae55a5fc2a2f690bd0710151a" gracePeriod=30 Sep 09 17:42:52.353247 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:52.353213 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb"] Sep 09 17:42:52.353512 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:52.353438 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="registry" containerID="cri-o://582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38" gracePeriod=30 Sep 09 17:42:52.353512 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:52.353491 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="ui" containerID="cri-o://83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e" gracePeriod=30 Sep 09 17:42:52.353688 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:52.353508 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="online" containerID="cri-o://02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8" gracePeriod=30 Sep 09 17:42:52.353688 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:52.353505 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="offline" containerID="cri-o://a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22" gracePeriod=30 Sep 09 17:42:52.521194 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:52.521095 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 17:42:53.201111 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:53.201085 2553 generic.go:356] "Generic (PLEG): container finished" podID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerID="83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e" exitCode=0 Sep 09 17:42:53.201111 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:53.201109 2553 generic.go:356] "Generic (PLEG): container finished" podID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerID="02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8" exitCode=0 Sep 09 17:42:53.201455 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:53.201152 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerDied","Data":"83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e"} Sep 09 17:42:53.201455 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:53.201195 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerDied","Data":"02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8"} Sep 09 17:42:53.203882 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:53.203856 2553 generic.go:356] "Generic (PLEG): container finished" podID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerID="e98b5a0f5b615e6a2d7f7db5b2dad056f399f4d8a325cd29af6cf892cdf1508a" exitCode=0 Sep 09 17:42:53.203882 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:53.203873 2553 generic.go:356] "Generic (PLEG): container finished" podID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerID="833bc352856e10c104f2ebbc35f71a6c7ec8e03b578a300b24f67491ad8b00c1" exitCode=0 Sep 09 17:42:53.204010 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:53.203923 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerDied","Data":"e98b5a0f5b615e6a2d7f7db5b2dad056f399f4d8a325cd29af6cf892cdf1508a"} Sep 09 17:42:53.204010 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:53.203951 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerDied","Data":"833bc352856e10c104f2ebbc35f71a6c7ec8e03b578a300b24f67491ad8b00c1"} Sep 09 17:42:55.940925 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:55.940879 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Sep 09 17:42:59.269342 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:42:59.269303 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 09 17:43:01.696660 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:01.696591 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 09 17:43:02.521485 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:02.521444 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 17:43:05.940464 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:05.940415 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Sep 09 17:43:09.269403 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:09.269359 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 09 17:43:11.697078 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:11.697037 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 09 17:43:12.521581 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:12.521541 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 17:43:12.521762 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:12.521697 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:43:15.940678 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:15.940613 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Sep 09 17:43:15.941094 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:15.940788 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:43:19.268526 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:19.268491 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 09 17:43:19.268923 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:19.268584 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:43:21.697086 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:21.697039 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Sep 09 17:43:21.697473 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:21.697211 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:43:22.312438 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.312412 2553 generic.go:356] "Generic (PLEG): container finished" podID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerID="cd89d259001d74c18b96210cb1cb49c36fc1272ae55a5fc2a2f690bd0710151a" exitCode=137 Sep 09 17:43:22.312539 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.312484 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerDied","Data":"cd89d259001d74c18b96210cb1cb49c36fc1272ae55a5fc2a2f690bd0710151a"} Sep 09 17:43:22.521711 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.521644 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 17:43:22.839778 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.839754 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:43:22.888224 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.888193 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-online-tls\") pod \"0f7a7eca-177b-4990-b8a0-6034db11ce80\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " Sep 09 17:43:22.888403 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.888279 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0f7a7eca-177b-4990-b8a0-6034db11ce80-registry-tls\") pod \"0f7a7eca-177b-4990-b8a0-6034db11ce80\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " Sep 09 17:43:22.888403 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.888308 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2xk6\" (UniqueName: \"kubernetes.io/projected/0f7a7eca-177b-4990-b8a0-6034db11ce80-kube-api-access-k2xk6\") pod \"0f7a7eca-177b-4990-b8a0-6034db11ce80\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " Sep 09 17:43:22.888403 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.888365 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7a7eca-177b-4990-b8a0-6034db11ce80-feast-data\") pod \"0f7a7eca-177b-4990-b8a0-6034db11ce80\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " Sep 09 17:43:22.888403 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.888396 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-offline-tls\") pod \"0f7a7eca-177b-4990-b8a0-6034db11ce80\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " Sep 09 17:43:22.888648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.888455 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-ui-tls\") pod \"0f7a7eca-177b-4990-b8a0-6034db11ce80\" (UID: \"0f7a7eca-177b-4990-b8a0-6034db11ce80\") " Sep 09 17:43:22.888715 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.888653 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7a7eca-177b-4990-b8a0-6034db11ce80-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0f7a7eca-177b-4990-b8a0-6034db11ce80" (UID: "0f7a7eca-177b-4990-b8a0-6034db11ce80"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:43:22.888918 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.888877 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7a7eca-177b-4990-b8a0-6034db11ce80-feast-data" (OuterVolumeSpecName: "feast-data") pod "0f7a7eca-177b-4990-b8a0-6034db11ce80" (UID: "0f7a7eca-177b-4990-b8a0-6034db11ce80"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:43:22.890583 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.890551 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-online-tls" (OuterVolumeSpecName: "online-tls") pod "0f7a7eca-177b-4990-b8a0-6034db11ce80" (UID: "0f7a7eca-177b-4990-b8a0-6034db11ce80"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:43:22.890583 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.890567 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "0f7a7eca-177b-4990-b8a0-6034db11ce80" (UID: "0f7a7eca-177b-4990-b8a0-6034db11ce80"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:43:22.890769 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.890589 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "0f7a7eca-177b-4990-b8a0-6034db11ce80" (UID: "0f7a7eca-177b-4990-b8a0-6034db11ce80"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:43:22.890921 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.890906 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7a7eca-177b-4990-b8a0-6034db11ce80-kube-api-access-k2xk6" (OuterVolumeSpecName: "kube-api-access-k2xk6") pod "0f7a7eca-177b-4990-b8a0-6034db11ce80" (UID: "0f7a7eca-177b-4990-b8a0-6034db11ce80"). InnerVolumeSpecName "kube-api-access-k2xk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:43:22.981493 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.981472 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:43:22.989149 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.989130 2553 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-offline-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:22.989220 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.989152 2553 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-ui-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:22.989220 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.989161 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0f7a7eca-177b-4990-b8a0-6034db11ce80-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:22.989220 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.989168 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0f7a7eca-177b-4990-b8a0-6034db11ce80-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:22.989220 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.989177 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2xk6\" (UniqueName: \"kubernetes.io/projected/0f7a7eca-177b-4990-b8a0-6034db11ce80-kube-api-access-k2xk6\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:22.989220 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:22.989190 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7a7eca-177b-4990-b8a0-6034db11ce80-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:23.090293 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.090231 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-online-tls\") pod \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " Sep 09 17:43:23.090399 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.090299 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9642b6-12a2-4a9d-943a-c00ddb31583f-feast-data\") pod \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " Sep 09 17:43:23.090468 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.090417 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-registry-tls\") pod \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " Sep 09 17:43:23.090468 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.090444 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-offline-tls\") pod \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " Sep 09 17:43:23.090577 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.090484 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-ui-tls\") pod \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " Sep 09 17:43:23.090577 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.090518 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m7sh\" (UniqueName: \"kubernetes.io/projected/dd9642b6-12a2-4a9d-943a-c00ddb31583f-kube-api-access-2m7sh\") pod \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\" (UID: \"dd9642b6-12a2-4a9d-943a-c00ddb31583f\") " Sep 09 17:43:23.090826 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.090801 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9642b6-12a2-4a9d-943a-c00ddb31583f-feast-data" (OuterVolumeSpecName: "feast-data") pod "dd9642b6-12a2-4a9d-943a-c00ddb31583f" (UID: "dd9642b6-12a2-4a9d-943a-c00ddb31583f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:43:23.092517 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.092483 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "dd9642b6-12a2-4a9d-943a-c00ddb31583f" (UID: "dd9642b6-12a2-4a9d-943a-c00ddb31583f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:43:23.092602 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.092546 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "dd9642b6-12a2-4a9d-943a-c00ddb31583f" (UID: "dd9642b6-12a2-4a9d-943a-c00ddb31583f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:43:23.092602 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.092556 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9642b6-12a2-4a9d-943a-c00ddb31583f-kube-api-access-2m7sh" (OuterVolumeSpecName: "kube-api-access-2m7sh") pod "dd9642b6-12a2-4a9d-943a-c00ddb31583f" (UID: "dd9642b6-12a2-4a9d-943a-c00ddb31583f"). InnerVolumeSpecName "kube-api-access-2m7sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:43:23.092783 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.092763 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-online-tls" (OuterVolumeSpecName: "online-tls") pod "dd9642b6-12a2-4a9d-943a-c00ddb31583f" (UID: "dd9642b6-12a2-4a9d-943a-c00ddb31583f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:43:23.092783 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.092775 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dd9642b6-12a2-4a9d-943a-c00ddb31583f" (UID: "dd9642b6-12a2-4a9d-943a-c00ddb31583f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:43:23.191497 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.191469 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9642b6-12a2-4a9d-943a-c00ddb31583f-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:23.191497 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.191497 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:23.191644 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.191507 2553 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-offline-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:23.191644 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.191515 2553 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-ui-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:23.191644 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.191523 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m7sh\" (UniqueName: \"kubernetes.io/projected/dd9642b6-12a2-4a9d-943a-c00ddb31583f-kube-api-access-2m7sh\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:23.191644 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.191531 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dd9642b6-12a2-4a9d-943a-c00ddb31583f-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:23.318207 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.318185 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" Sep 09 17:43:23.318207 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.318189 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz" event={"ID":"0f7a7eca-177b-4990-b8a0-6034db11ce80","Type":"ContainerDied","Data":"a00fc82e36e5af711b72a9f90f221019bb0592e0188a258aa8af0256c729863f"} Sep 09 17:43:23.318406 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.318242 2553 scope.go:117] "RemoveContainer" containerID="e98b5a0f5b615e6a2d7f7db5b2dad056f399f4d8a325cd29af6cf892cdf1508a" Sep 09 17:43:23.321301 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.321273 2553 generic.go:356] "Generic (PLEG): container finished" podID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerID="a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22" exitCode=137 Sep 09 17:43:23.321301 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.321297 2553 generic.go:356] "Generic (PLEG): container finished" podID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerID="582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38" exitCode=137 Sep 09 17:43:23.321478 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.321320 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerDied","Data":"a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22"} Sep 09 17:43:23.321478 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.321374 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerDied","Data":"582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38"} Sep 09 17:43:23.321478 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.321397 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" event={"ID":"dd9642b6-12a2-4a9d-943a-c00ddb31583f","Type":"ContainerDied","Data":"ec18287c98570c5266ad6932981c69f6d8fa2e37548c8df51bcce6736133d1a4"} Sep 09 17:43:23.321478 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.321334 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb" Sep 09 17:43:23.327764 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.327749 2553 scope.go:117] "RemoveContainer" containerID="cd89d259001d74c18b96210cb1cb49c36fc1272ae55a5fc2a2f690bd0710151a" Sep 09 17:43:23.335749 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.335729 2553 scope.go:117] "RemoveContainer" containerID="833bc352856e10c104f2ebbc35f71a6c7ec8e03b578a300b24f67491ad8b00c1" Sep 09 17:43:23.342421 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.342370 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz"] Sep 09 17:43:23.344441 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.344422 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-94qhz"] Sep 09 17:43:23.344685 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.344668 2553 scope.go:117] "RemoveContainer" containerID="3584db2e2b90e78538b8ab15cc2dbfd9a39dc2b8b20a6af74e8f563bfe35c0ab" Sep 09 17:43:23.351600 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.351575 2553 scope.go:117] "RemoveContainer" containerID="06899502ca059f666b3e6a67fa7b248367212d80d36318fe3c057a8a64c91049" Sep 09 17:43:23.357364 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.357345 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb"] Sep 09 17:43:23.358405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.358392 2553 scope.go:117] "RemoveContainer" containerID="83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e" Sep 09 17:43:23.360833 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.360813 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-xqqgb"] Sep 09 17:43:23.365263 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.365245 2553 scope.go:117] "RemoveContainer" containerID="a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22" Sep 09 17:43:23.371879 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.371865 2553 scope.go:117] "RemoveContainer" containerID="02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8" Sep 09 17:43:23.378364 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.378347 2553 scope.go:117] "RemoveContainer" containerID="582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38" Sep 09 17:43:23.384945 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.384928 2553 scope.go:117] "RemoveContainer" containerID="adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62" Sep 09 17:43:23.391201 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.391186 2553 scope.go:117] "RemoveContainer" containerID="159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3" Sep 09 17:43:23.397422 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.397407 2553 scope.go:117] "RemoveContainer" containerID="83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e" Sep 09 17:43:23.397683 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:43:23.397662 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e\": container with ID starting with 83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e not found: ID does not exist" containerID="83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e" Sep 09 17:43:23.397741 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.397695 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e"} err="failed to get container status \"83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e\": rpc error: code = NotFound desc = could not find container \"83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e\": container with ID starting with 83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e not found: ID does not exist" Sep 09 17:43:23.397741 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.397717 2553 scope.go:117] "RemoveContainer" containerID="a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22" Sep 09 17:43:23.397942 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:43:23.397927 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22\": container with ID starting with a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22 not found: ID does not exist" containerID="a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22" Sep 09 17:43:23.397988 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.397946 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22"} err="failed to get container status \"a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22\": rpc error: code = NotFound desc = could not find container \"a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22\": container with ID starting with a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22 not found: ID does not exist" Sep 09 17:43:23.397988 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.397958 2553 scope.go:117] "RemoveContainer" containerID="02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8" Sep 09 17:43:23.398154 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:43:23.398141 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8\": container with ID starting with 02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8 not found: ID does not exist" containerID="02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8" Sep 09 17:43:23.398191 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.398157 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8"} err="failed to get container status \"02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8\": rpc error: code = NotFound desc = could not find container \"02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8\": container with ID starting with 02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8 not found: ID does not exist" Sep 09 17:43:23.398191 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.398182 2553 scope.go:117] "RemoveContainer" containerID="582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38" Sep 09 17:43:23.398375 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:43:23.398360 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38\": container with ID starting with 582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38 not found: ID does not exist" containerID="582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38" Sep 09 17:43:23.398413 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.398377 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38"} err="failed to get container status \"582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38\": rpc error: code = NotFound desc = could not find container \"582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38\": container with ID starting with 582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38 not found: ID does not exist" Sep 09 17:43:23.398413 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.398388 2553 scope.go:117] "RemoveContainer" containerID="adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62" Sep 09 17:43:23.398573 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:43:23.398558 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62\": container with ID starting with adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62 not found: ID does not exist" containerID="adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62" Sep 09 17:43:23.398616 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.398577 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62"} err="failed to get container status \"adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62\": rpc error: code = NotFound desc = could not find container \"adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62\": container with ID starting with adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62 not found: ID does not exist" Sep 09 17:43:23.398616 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.398590 2553 scope.go:117] "RemoveContainer" containerID="159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3" Sep 09 17:43:23.398836 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:43:23.398807 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3\": container with ID starting with 159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3 not found: ID does not exist" containerID="159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3" Sep 09 17:43:23.398927 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.398841 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3"} err="failed to get container status \"159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3\": rpc error: code = NotFound desc = could not find container \"159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3\": container with ID starting with 159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3 not found: ID does not exist" Sep 09 17:43:23.398927 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.398862 2553 scope.go:117] "RemoveContainer" containerID="83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e" Sep 09 17:43:23.399097 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.399078 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e"} err="failed to get container status \"83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e\": rpc error: code = NotFound desc = could not find container \"83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e\": container with ID starting with 83ff29844d0f38ceac11cf80309ceaaca52dd87c79598d17e9289a830341290e not found: ID does not exist" Sep 09 17:43:23.399149 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.399098 2553 scope.go:117] "RemoveContainer" containerID="a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22" Sep 09 17:43:23.399310 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.399293 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22"} err="failed to get container status \"a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22\": rpc error: code = NotFound desc = could not find container \"a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22\": container with ID starting with a6943b13e61d3e50998a69c6e4315598a88d3eb52009a5fb5cce8d8aeff42a22 not found: ID does not exist" Sep 09 17:43:23.399380 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.399312 2553 scope.go:117] "RemoveContainer" containerID="02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8" Sep 09 17:43:23.399555 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.399537 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8"} err="failed to get container status \"02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8\": rpc error: code = NotFound desc = could not find container \"02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8\": container with ID starting with 02a1b8a4ec9431f667ff90cce7ca549088f4accf065ec40d6b33d26eb24a6fb8 not found: ID does not exist" Sep 09 17:43:23.399613 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.399555 2553 scope.go:117] "RemoveContainer" containerID="582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38" Sep 09 17:43:23.399766 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.399745 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38"} err="failed to get container status \"582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38\": rpc error: code = NotFound desc = could not find container \"582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38\": container with ID starting with 582717ac3c094e6d9e9665f5a3e891b5b1c56b3b9f1d9b6d4089abb09481cd38 not found: ID does not exist" Sep 09 17:43:23.399821 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.399768 2553 scope.go:117] "RemoveContainer" containerID="adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62" Sep 09 17:43:23.399962 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.399944 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62"} err="failed to get container status \"adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62\": rpc error: code = NotFound desc = could not find container \"adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62\": container with ID starting with adcfc45ee283927647daff6f258554f6d0a61bb36d580beee58a8019a62bcc62 not found: ID does not exist" Sep 09 17:43:23.400036 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.399963 2553 scope.go:117] "RemoveContainer" containerID="159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3" Sep 09 17:43:23.400166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:23.400149 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3"} err="failed to get container status \"159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3\": rpc error: code = NotFound desc = could not find container \"159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3\": container with ID starting with 159b814ddaba5b4bdcb06fc40be90731fa579d585e7f49d839ac4df1e9c8b8d3 not found: ID does not exist" Sep 09 17:43:24.348870 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:24.348839 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" path="/var/lib/kubelet/pods/0f7a7eca-177b-4990-b8a0-6034db11ce80/volumes" Sep 09 17:43:24.349359 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:24.349345 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" path="/var/lib/kubelet/pods/dd9642b6-12a2-4a9d-943a-c00ddb31583f/volumes" Sep 09 17:43:25.332031 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:25.331999 2553 generic.go:356] "Generic (PLEG): container finished" podID="7afd2753-0fa9-438f-a6dc-b6be29e0ac2d" containerID="3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c" exitCode=1 Sep 09 17:43:25.332197 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:25.332037 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" event={"ID":"7afd2753-0fa9-438f-a6dc-b6be29e0ac2d","Type":"ContainerDied","Data":"3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c"} Sep 09 17:43:25.332325 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:25.332310 2553 scope.go:117] "RemoveContainer" containerID="3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c" Sep 09 17:43:26.338186 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:26.338154 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" event={"ID":"7afd2753-0fa9-438f-a6dc-b6be29e0ac2d","Type":"ContainerStarted","Data":"e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8"} Sep 09 17:43:26.338546 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:26.338353 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" Sep 09 17:43:37.343900 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:37.343864 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" Sep 09 17:43:41.052357 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.052322 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s"] Sep 09 17:43:41.052838 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.052575 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" podUID="7afd2753-0fa9-438f-a6dc-b6be29e0ac2d" containerName="manager" containerID="cri-o://e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8" gracePeriod=10 Sep 09 17:43:41.293671 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.293648 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" Sep 09 17:43:41.390301 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.390213 2553 generic.go:356] "Generic (PLEG): container finished" podID="7afd2753-0fa9-438f-a6dc-b6be29e0ac2d" containerID="e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8" exitCode=0 Sep 09 17:43:41.390301 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.390263 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" Sep 09 17:43:41.390470 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.390303 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" event={"ID":"7afd2753-0fa9-438f-a6dc-b6be29e0ac2d","Type":"ContainerDied","Data":"e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8"} Sep 09 17:43:41.390470 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.390357 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s" event={"ID":"7afd2753-0fa9-438f-a6dc-b6be29e0ac2d","Type":"ContainerDied","Data":"ce77da7ada73731f5bf68efdfa49c28a078869d3008008b1e94ab0670148ca3d"} Sep 09 17:43:41.390470 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.390380 2553 scope.go:117] "RemoveContainer" containerID="e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8" Sep 09 17:43:41.397686 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.397671 2553 scope.go:117] "RemoveContainer" containerID="3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c" Sep 09 17:43:41.404522 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.404503 2553 scope.go:117] "RemoveContainer" containerID="e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8" Sep 09 17:43:41.404814 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:43:41.404787 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8\": container with ID starting with e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8 not found: ID does not exist" containerID="e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8" Sep 09 17:43:41.404893 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.404827 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8"} err="failed to get container status \"e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8\": rpc error: code = NotFound desc = could not find container \"e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8\": container with ID starting with e3b1e3941c45a3a7b4b97974a74718ec0c047b6d0316eb57d818658d344ef0e8 not found: ID does not exist" Sep 09 17:43:41.404893 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.404849 2553 scope.go:117] "RemoveContainer" containerID="3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c" Sep 09 17:43:41.405122 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:43:41.405105 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c\": container with ID starting with 3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c not found: ID does not exist" containerID="3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c" Sep 09 17:43:41.405169 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.405128 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c"} err="failed to get container status \"3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c\": rpc error: code = NotFound desc = could not find container \"3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c\": container with ID starting with 3bcbb2fa37afab1366c412a977ee927cdb69c809dfb874e773ac28f2e5777d9c not found: ID does not exist" Sep 09 17:43:41.422420 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.422398 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crl5g\" (UniqueName: \"kubernetes.io/projected/7afd2753-0fa9-438f-a6dc-b6be29e0ac2d-kube-api-access-crl5g\") pod \"7afd2753-0fa9-438f-a6dc-b6be29e0ac2d\" (UID: \"7afd2753-0fa9-438f-a6dc-b6be29e0ac2d\") " Sep 09 17:43:41.424367 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.424346 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afd2753-0fa9-438f-a6dc-b6be29e0ac2d-kube-api-access-crl5g" (OuterVolumeSpecName: "kube-api-access-crl5g") pod "7afd2753-0fa9-438f-a6dc-b6be29e0ac2d" (UID: "7afd2753-0fa9-438f-a6dc-b6be29e0ac2d"). InnerVolumeSpecName "kube-api-access-crl5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:43:41.524049 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.524016 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crl5g\" (UniqueName: \"kubernetes.io/projected/7afd2753-0fa9-438f-a6dc-b6be29e0ac2d-kube-api-access-crl5g\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:43:41.710172 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.710142 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s"] Sep 09 17:43:41.713732 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:41.713708 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-95w6s"] Sep 09 17:43:42.346099 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:42.346068 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7afd2753-0fa9-438f-a6dc-b6be29e0ac2d" path="/var/lib/kubelet/pods/7afd2753-0fa9-438f-a6dc-b6be29e0ac2d/volumes" Sep 09 17:43:46.008797 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.008764 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx"] Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009039 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="registry" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009049 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="registry" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009058 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="ui" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009064 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="ui" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009072 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="offline" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009077 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="offline" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009088 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="online" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009093 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="online" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009098 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="ui" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009103 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="ui" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009109 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="offline" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009113 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="offline" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009119 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7afd2753-0fa9-438f-a6dc-b6be29e0ac2d" containerName="manager" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009124 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7afd2753-0fa9-438f-a6dc-b6be29e0ac2d" containerName="manager" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009130 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="feast-init" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009134 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="feast-init" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009141 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="online" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009146 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="online" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009153 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="feast-apply" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009159 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="feast-apply" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009164 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="feast-init" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009168 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="feast-init" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009174 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="feast-apply" Sep 09 17:43:46.009166 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009178 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="feast-apply" Sep 09 17:43:46.009922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009220 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="registry" Sep 09 17:43:46.009922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009229 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="offline" Sep 09 17:43:46.009922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009236 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="online" Sep 09 17:43:46.009922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009242 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7afd2753-0fa9-438f-a6dc-b6be29e0ac2d" containerName="manager" Sep 09 17:43:46.009922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009247 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="online" Sep 09 17:43:46.009922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009252 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="offline" Sep 09 17:43:46.009922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009260 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7afd2753-0fa9-438f-a6dc-b6be29e0ac2d" containerName="manager" Sep 09 17:43:46.009922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009266 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd9642b6-12a2-4a9d-943a-c00ddb31583f" containerName="ui" Sep 09 17:43:46.009922 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.009271 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f7a7eca-177b-4990-b8a0-6034db11ce80" containerName="ui" Sep 09 17:43:46.013242 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.013224 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx" Sep 09 17:43:46.016994 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.016966 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-4pk25\"" Sep 09 17:43:46.017120 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.017011 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 09 17:43:46.018042 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.018027 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 09 17:43:46.031049 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.031028 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx"] Sep 09 17:43:46.154360 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.154328 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx87f\" (UniqueName: \"kubernetes.io/projected/4c9df85c-6433-4b0f-b11f-0dece1b8cf3a-kube-api-access-tx87f\") pod \"feast-operator-controller-manager-689b877b4b-m6rmx\" (UID: \"4c9df85c-6433-4b0f-b11f-0dece1b8cf3a\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx" Sep 09 17:43:46.255245 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.255210 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx87f\" (UniqueName: \"kubernetes.io/projected/4c9df85c-6433-4b0f-b11f-0dece1b8cf3a-kube-api-access-tx87f\") pod \"feast-operator-controller-manager-689b877b4b-m6rmx\" (UID: \"4c9df85c-6433-4b0f-b11f-0dece1b8cf3a\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx" Sep 09 17:43:46.264288 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.264242 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 09 17:43:46.274553 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.274534 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 09 17:43:46.284951 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.284922 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx87f\" (UniqueName: \"kubernetes.io/projected/4c9df85c-6433-4b0f-b11f-0dece1b8cf3a-kube-api-access-tx87f\") pod \"feast-operator-controller-manager-689b877b4b-m6rmx\" (UID: \"4c9df85c-6433-4b0f-b11f-0dece1b8cf3a\") " pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx" Sep 09 17:43:46.325026 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.325009 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-4pk25\"" Sep 09 17:43:46.333062 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.333047 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx" Sep 09 17:43:46.603832 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:46.603666 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx"] Sep 09 17:43:46.606248 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:43:46.606223 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9df85c_6433_4b0f_b11f_0dece1b8cf3a.slice/crio-7501414b519621b280d7d5e66c9cbc3754f11540db119d50d56cd9fc8ae7a7c3 WatchSource:0}: Error finding container 7501414b519621b280d7d5e66c9cbc3754f11540db119d50d56cd9fc8ae7a7c3: Status 404 returned error can't find the container with id 7501414b519621b280d7d5e66c9cbc3754f11540db119d50d56cd9fc8ae7a7c3 Sep 09 17:43:47.413483 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:47.413448 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx" event={"ID":"4c9df85c-6433-4b0f-b11f-0dece1b8cf3a","Type":"ContainerStarted","Data":"842526a480ef3df3a7beb3172e275c44a482a47ad4303eed8f49dc8ff66a8399"} Sep 09 17:43:47.413877 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:47.413498 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx" event={"ID":"4c9df85c-6433-4b0f-b11f-0dece1b8cf3a","Type":"ContainerStarted","Data":"7501414b519621b280d7d5e66c9cbc3754f11540db119d50d56cd9fc8ae7a7c3"} Sep 09 17:43:47.413877 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:47.413563 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx" Sep 09 17:43:47.430748 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:47.430701 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx" podStartSLOduration=2.430687458 podStartE2EDuration="2.430687458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:43:47.429496523 +0000 UTC m=+1501.669402876" watchObservedRunningTime="2026-09-09 17:43:47.430687458 +0000 UTC m=+1501.670593840" Sep 09 17:43:58.418881 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:43:58.418854 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-689b877b4b-m6rmx" Sep 09 17:44:03.797643 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.795517 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk"] Sep 09 17:44:03.797643 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.796245 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7afd2753-0fa9-438f-a6dc-b6be29e0ac2d" containerName="manager" Sep 09 17:44:03.797643 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.796265 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7afd2753-0fa9-438f-a6dc-b6be29e0ac2d" containerName="manager" Sep 09 17:44:03.802006 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.801982 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.805527 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.805505 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 17:44:03.805676 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.805557 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 17:44:03.806187 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.806166 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 17:44:03.806289 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.806187 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-5v8fg\"" Sep 09 17:44:03.806289 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.806165 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 17:44:03.806289 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.806171 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 17:44:03.806289 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.806171 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 17:44:03.810758 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.810735 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk"] Sep 09 17:44:03.885120 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.885091 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.885261 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.885165 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.885261 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.885190 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59f980bd-b9ee-44be-ba9c-aeb829a96598-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.885261 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.885212 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.885386 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.885260 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm9m6\" (UniqueName: \"kubernetes.io/projected/59f980bd-b9ee-44be-ba9c-aeb829a96598-kube-api-access-nm9m6\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.885386 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.885293 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.986038 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.986003 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm9m6\" (UniqueName: \"kubernetes.io/projected/59f980bd-b9ee-44be-ba9c-aeb829a96598-kube-api-access-nm9m6\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.986038 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.986039 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.986283 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.986057 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.986283 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.986100 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.986283 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.986117 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59f980bd-b9ee-44be-ba9c-aeb829a96598-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.986283 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.986134 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.986283 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:44:03.986162 2553 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 09 17:44:03.986283 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:44:03.986226 2553 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 17:44:03.986283 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:44:03.986247 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-online-tls podName:59f980bd-b9ee-44be-ba9c-aeb829a96598 nodeName:}" failed. No retries permitted until 2026-09-09 17:44:04.486224889 +0000 UTC m=+1518.726131224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-online-tls") pod "feast-simple-feast-setup-57576b8fcf-l99pk" (UID: "59f980bd-b9ee-44be-ba9c-aeb829a96598") : secret "feast-simple-feast-setup-online-tls" not found Sep 09 17:44:03.986283 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:44:03.986275 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-ui-tls podName:59f980bd-b9ee-44be-ba9c-aeb829a96598 nodeName:}" failed. No retries permitted until 2026-09-09 17:44:04.48625765 +0000 UTC m=+1518.726163997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-l99pk" (UID: "59f980bd-b9ee-44be-ba9c-aeb829a96598") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 17:44:03.986607 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.986539 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59f980bd-b9ee-44be-ba9c-aeb829a96598-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.988469 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.988438 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:03.988576 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:03.988493 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:04.000537 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:04.000516 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm9m6\" (UniqueName: \"kubernetes.io/projected/59f980bd-b9ee-44be-ba9c-aeb829a96598-kube-api-access-nm9m6\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:04.490186 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:04.490157 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:04.490379 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:04.490221 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:04.492534 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:04.492504 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:04.492660 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:04.492580 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l99pk\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:04.721295 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:04.721266 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:04.999616 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:04.999581 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk"] Sep 09 17:44:05.001994 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:44:05.001970 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59f980bd_b9ee_44be_ba9c_aeb829a96598.slice/crio-d4c9ec93f9465599b2cdc3a61e55262fd9cd322c8833be6b716cfdaba8618153 WatchSource:0}: Error finding container d4c9ec93f9465599b2cdc3a61e55262fd9cd322c8833be6b716cfdaba8618153: Status 404 returned error can't find the container with id d4c9ec93f9465599b2cdc3a61e55262fd9cd322c8833be6b716cfdaba8618153 Sep 09 17:44:05.477490 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:05.477452 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerStarted","Data":"ef72f96de4bbadb2f1773d6a469a75b1a5b346a1b29fd1928d6ba8f9891fd7cd"} Sep 09 17:44:05.477726 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:05.477516 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerStarted","Data":"d4c9ec93f9465599b2cdc3a61e55262fd9cd322c8833be6b716cfdaba8618153"} Sep 09 17:44:10.496963 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:10.496927 2553 generic.go:356] "Generic (PLEG): container finished" podID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerID="ef72f96de4bbadb2f1773d6a469a75b1a5b346a1b29fd1928d6ba8f9891fd7cd" exitCode=0 Sep 09 17:44:10.497318 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:10.496990 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerDied","Data":"ef72f96de4bbadb2f1773d6a469a75b1a5b346a1b29fd1928d6ba8f9891fd7cd"} Sep 09 17:44:11.502409 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:11.502374 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerStarted","Data":"e75d15022dcdb76bddbddf26b0e0996ae03ffba61e045ea6ee4a2d96e91b9109"} Sep 09 17:44:16.521953 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:16.521867 2553 generic.go:356] "Generic (PLEG): container finished" podID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerID="e75d15022dcdb76bddbddf26b0e0996ae03ffba61e045ea6ee4a2d96e91b9109" exitCode=0 Sep 09 17:44:16.521953 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:16.521918 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerDied","Data":"e75d15022dcdb76bddbddf26b0e0996ae03ffba61e045ea6ee4a2d96e91b9109"} Sep 09 17:44:17.532220 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:17.532172 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerStarted","Data":"b14603af5fe81ce0ffeac1d908401209ae150058d647b9e78eaa5d6ab1d7fae2"} Sep 09 17:44:17.532723 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:17.532238 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerStarted","Data":"f4cee327ad38809dc57b1d19a7fa1b95966bdf16a3f82849f702bddce261edff"} Sep 09 17:44:17.532723 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:17.532258 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerStarted","Data":"9a939723fb527681a097f14976d06fec57705f59a58c6789222fc7c796a1e67d"} Sep 09 17:44:17.532723 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:17.532277 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerStarted","Data":"c766f8d4749d66a1314ee4a38e266d26bfd0a930789f77a076f829190cdb06e2"} Sep 09 17:44:17.558422 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:17.558358 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podStartSLOduration=14.558335736 podStartE2EDuration="14.558335736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:44:17.555680113 +0000 UTC m=+1531.795586473" watchObservedRunningTime="2026-09-09 17:44:17.558335736 +0000 UTC m=+1531.798242090" Sep 09 17:44:19.721859 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:19.721794 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:19.721859 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:19.721859 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:19.722404 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:19.721879 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:19.722404 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:19.721897 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:19.723752 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:19.723719 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused" Sep 09 17:44:19.723998 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:19.723747 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 09 17:44:19.724089 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:19.723772 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 17:44:19.724089 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:19.723789 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 09 17:44:22.722183 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:22.722125 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 09 17:44:22.722681 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:22.722125 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 09 17:44:22.722681 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:22.722138 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 17:44:22.722681 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:22.722298 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused" Sep 09 17:44:24.721766 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:24.721725 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:24.722162 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:24.721797 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:24.722162 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:24.721833 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:24.722162 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:24.721848 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:25.722937 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:25.722907 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:25.723378 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:25.723077 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:25.723378 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:25.723280 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:25.727844 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:25.727828 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:26.576605 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:26.576578 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:26.576822 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:26.576656 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:26.576822 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:26.576690 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:44:26.579254 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:44:26.579231 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:45:29.360567 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.360535 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk"] Sep 09 17:45:29.361026 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.360782 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="registry" containerID="cri-o://c766f8d4749d66a1314ee4a38e266d26bfd0a930789f77a076f829190cdb06e2" gracePeriod=30 Sep 09 17:45:29.361026 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.360832 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="online" containerID="cri-o://9a939723fb527681a097f14976d06fec57705f59a58c6789222fc7c796a1e67d" gracePeriod=30 Sep 09 17:45:29.361026 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.360867 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="ui" containerID="cri-o://b14603af5fe81ce0ffeac1d908401209ae150058d647b9e78eaa5d6ab1d7fae2" gracePeriod=30 Sep 09 17:45:29.361026 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.360833 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="offline" containerID="cri-o://f4cee327ad38809dc57b1d19a7fa1b95966bdf16a3f82849f702bddce261edff" gracePeriod=30 Sep 09 17:45:29.784253 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.784214 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf"] Sep 09 17:45:29.788586 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.788562 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.791212 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.791189 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-9gxbc\"" Sep 09 17:45:29.798365 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.798339 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf"] Sep 09 17:45:29.862075 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.862042 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.862075 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.862077 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.862315 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.862113 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d806f0c6-ed63-4345-87e1-4ee6b3742339-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.862315 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.862181 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.862315 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.862248 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctr7h\" (UniqueName: \"kubernetes.io/projected/d806f0c6-ed63-4345-87e1-4ee6b3742339-kube-api-access-ctr7h\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.862315 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.862274 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.963458 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.963412 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.963668 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.963466 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.963668 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.963523 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d806f0c6-ed63-4345-87e1-4ee6b3742339-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.963668 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.963568 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.963668 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:45:29.963576 2553 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 09 17:45:29.963668 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.963640 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctr7h\" (UniqueName: \"kubernetes.io/projected/d806f0c6-ed63-4345-87e1-4ee6b3742339-kube-api-access-ctr7h\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.963668 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:45:29.963667 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-online-tls podName:d806f0c6-ed63-4345-87e1-4ee6b3742339 nodeName:}" failed. No retries permitted until 2026-09-09 17:45:30.46364542 +0000 UTC m=+1604.703551773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-online-tls") pod "feast-simple-feast-setup-57576b8fcf-wk8jf" (UID: "d806f0c6-ed63-4345-87e1-4ee6b3742339") : secret "feast-simple-feast-setup-online-tls" not found Sep 09 17:45:29.964011 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.963707 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.964011 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:45:29.963747 2553 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 09 17:45:29.964011 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:45:29.963802 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-registry-tls podName:d806f0c6-ed63-4345-87e1-4ee6b3742339 nodeName:}" failed. No retries permitted until 2026-09-09 17:45:30.463784865 +0000 UTC m=+1604.703691224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-wk8jf" (UID: "d806f0c6-ed63-4345-87e1-4ee6b3742339") : secret "feast-simple-feast-setup-registry-tls" not found Sep 09 17:45:29.964171 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.964021 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d806f0c6-ed63-4345-87e1-4ee6b3742339-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.966135 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.966114 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.966252 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.966168 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:29.972219 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:29.972191 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctr7h\" (UniqueName: \"kubernetes.io/projected/d806f0c6-ed63-4345-87e1-4ee6b3742339-kube-api-access-ctr7h\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:30.467189 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:30.467161 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:30.467791 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:30.467765 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:30.470731 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:30.470700 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:30.474443 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:30.474421 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-wk8jf\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:30.699291 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:30.699263 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:30.808262 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:30.808234 2553 generic.go:356] "Generic (PLEG): container finished" podID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerID="b14603af5fe81ce0ffeac1d908401209ae150058d647b9e78eaa5d6ab1d7fae2" exitCode=0 Sep 09 17:45:30.808262 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:30.808257 2553 generic.go:356] "Generic (PLEG): container finished" podID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerID="9a939723fb527681a097f14976d06fec57705f59a58c6789222fc7c796a1e67d" exitCode=0 Sep 09 17:45:30.808461 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:30.808306 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerDied","Data":"b14603af5fe81ce0ffeac1d908401209ae150058d647b9e78eaa5d6ab1d7fae2"} Sep 09 17:45:30.808461 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:30.808350 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerDied","Data":"9a939723fb527681a097f14976d06fec57705f59a58c6789222fc7c796a1e67d"} Sep 09 17:45:30.971155 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:30.971129 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf"] Sep 09 17:45:30.972734 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:45:30.972705 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd806f0c6_ed63_4345_87e1_4ee6b3742339.slice/crio-6310e2aaa15e4625bdb00d4d36f4ab6940839a7ab48a405cb57537bc221ab12a WatchSource:0}: Error finding container 6310e2aaa15e4625bdb00d4d36f4ab6940839a7ab48a405cb57537bc221ab12a: Status 404 returned error can't find the container with id 6310e2aaa15e4625bdb00d4d36f4ab6940839a7ab48a405cb57537bc221ab12a Sep 09 17:45:31.814181 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:31.814146 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerStarted","Data":"d38a73e8b6bd6044ac42cc2deec5111e4c6d1eca514e8f2bbca082ff8c651ea9"} Sep 09 17:45:31.814181 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:31.814187 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerStarted","Data":"6310e2aaa15e4625bdb00d4d36f4ab6940839a7ab48a405cb57537bc221ab12a"} Sep 09 17:45:35.831070 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:35.831033 2553 generic.go:356] "Generic (PLEG): container finished" podID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerID="d38a73e8b6bd6044ac42cc2deec5111e4c6d1eca514e8f2bbca082ff8c651ea9" exitCode=0 Sep 09 17:45:35.831458 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:35.831107 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerDied","Data":"d38a73e8b6bd6044ac42cc2deec5111e4c6d1eca514e8f2bbca082ff8c651ea9"} Sep 09 17:45:36.577118 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:36.577069 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 09 17:45:36.577291 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:36.577077 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 17:45:36.838110 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:36.838028 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerStarted","Data":"abd24489c750c3d6ba8f62bb7cb20c973e1ee8f888d7f7844b9a968607750ebc"} Sep 09 17:45:41.859643 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:41.859583 2553 generic.go:356] "Generic (PLEG): container finished" podID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerID="abd24489c750c3d6ba8f62bb7cb20c973e1ee8f888d7f7844b9a968607750ebc" exitCode=0 Sep 09 17:45:41.860044 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:41.859658 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerDied","Data":"abd24489c750c3d6ba8f62bb7cb20c973e1ee8f888d7f7844b9a968607750ebc"} Sep 09 17:45:42.870088 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:42.870045 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerStarted","Data":"a0728036db57c2ca5ab240d392b162ef7b0c5127dbf603ef061aeda7248cf929"} Sep 09 17:45:42.870556 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:42.870106 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerStarted","Data":"db28c97f7a9c12fca0c9abddcded3883c5e0708fd42bd6108e638e31ed3c06eb"} Sep 09 17:45:42.870556 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:42.870125 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerStarted","Data":"199f37dfcbdc457f1de14ba8e07cd9e5ef6d249dbd48b2606b1f772b35352f20"} Sep 09 17:45:42.870556 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:42.870143 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerStarted","Data":"614d8d2030d1663e785e4ec2310486d4d35330d420faa205d3a34e96db4ba302"} Sep 09 17:45:42.895070 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:42.895014 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podStartSLOduration=13.894992299 podStartE2EDuration="13.894992299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:45:42.892650102 +0000 UTC m=+1617.132556481" watchObservedRunningTime="2026-09-09 17:45:42.894992299 +0000 UTC m=+1617.134898650" Sep 09 17:45:45.700109 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:45.700068 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:45.700566 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:45.700137 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:45.700566 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:45.700158 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:45.700566 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:45.700176 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:45.702146 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:45.702107 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Sep 09 17:45:45.702146 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:45.702126 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused" Sep 09 17:45:45.702337 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:45.702145 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 09 17:45:45.702337 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:45.702223 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 17:45:46.577438 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:46.577052 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 17:45:46.577438 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:46.577096 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 09 17:45:48.699985 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:48.699922 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused" Sep 09 17:45:48.699985 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:48.699965 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 09 17:45:48.700517 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:48.699989 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 17:45:48.700700 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:48.700669 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Sep 09 17:45:50.699878 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:50.699840 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:50.700336 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:50.699918 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:50.700336 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:50.699955 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:50.700336 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:50.699968 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:51.700697 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:51.700668 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:51.701110 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:51.700770 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:51.701110 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:51.700974 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:51.705303 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:51.705286 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:51.931411 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:51.931381 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:51.931605 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:51.931451 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:51.931766 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:51.931749 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:51.933999 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:51.933980 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:45:56.576928 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:56.576883 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 09 17:45:56.577341 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:56.576887 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 17:45:56.577341 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:56.577016 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:45:56.577341 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:56.577091 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:45:59.965646 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:59.965593 2553 generic.go:356] "Generic (PLEG): container finished" podID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerID="f4cee327ad38809dc57b1d19a7fa1b95966bdf16a3f82849f702bddce261edff" exitCode=137 Sep 09 17:45:59.965646 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:59.965648 2553 generic.go:356] "Generic (PLEG): container finished" podID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerID="c766f8d4749d66a1314ee4a38e266d26bfd0a930789f77a076f829190cdb06e2" exitCode=137 Sep 09 17:45:59.966017 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:59.965660 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerDied","Data":"f4cee327ad38809dc57b1d19a7fa1b95966bdf16a3f82849f702bddce261edff"} Sep 09 17:45:59.966017 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:45:59.965704 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerDied","Data":"c766f8d4749d66a1314ee4a38e266d26bfd0a930789f77a076f829190cdb06e2"} Sep 09 17:46:00.006745 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.006723 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:46:00.104955 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.104924 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-registry-tls\") pod \"59f980bd-b9ee-44be-ba9c-aeb829a96598\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " Sep 09 17:46:00.105133 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.104975 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59f980bd-b9ee-44be-ba9c-aeb829a96598-feast-data\") pod \"59f980bd-b9ee-44be-ba9c-aeb829a96598\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " Sep 09 17:46:00.105133 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.104998 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-online-tls\") pod \"59f980bd-b9ee-44be-ba9c-aeb829a96598\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " Sep 09 17:46:00.105133 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.105030 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-offline-tls\") pod \"59f980bd-b9ee-44be-ba9c-aeb829a96598\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " Sep 09 17:46:00.105133 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.105048 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm9m6\" (UniqueName: \"kubernetes.io/projected/59f980bd-b9ee-44be-ba9c-aeb829a96598-kube-api-access-nm9m6\") pod \"59f980bd-b9ee-44be-ba9c-aeb829a96598\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " Sep 09 17:46:00.105133 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.105071 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-ui-tls\") pod \"59f980bd-b9ee-44be-ba9c-aeb829a96598\" (UID: \"59f980bd-b9ee-44be-ba9c-aeb829a96598\") " Sep 09 17:46:00.105668 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.105610 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f980bd-b9ee-44be-ba9c-aeb829a96598-feast-data" (OuterVolumeSpecName: "feast-data") pod "59f980bd-b9ee-44be-ba9c-aeb829a96598" (UID: "59f980bd-b9ee-44be-ba9c-aeb829a96598"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:46:00.107357 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.107299 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-online-tls" (OuterVolumeSpecName: "online-tls") pod "59f980bd-b9ee-44be-ba9c-aeb829a96598" (UID: "59f980bd-b9ee-44be-ba9c-aeb829a96598"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:46:00.107467 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.107355 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "59f980bd-b9ee-44be-ba9c-aeb829a96598" (UID: "59f980bd-b9ee-44be-ba9c-aeb829a96598"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:46:00.107467 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.107400 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f980bd-b9ee-44be-ba9c-aeb829a96598-kube-api-access-nm9m6" (OuterVolumeSpecName: "kube-api-access-nm9m6") pod "59f980bd-b9ee-44be-ba9c-aeb829a96598" (UID: "59f980bd-b9ee-44be-ba9c-aeb829a96598"). InnerVolumeSpecName "kube-api-access-nm9m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:46:00.107562 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.107460 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "59f980bd-b9ee-44be-ba9c-aeb829a96598" (UID: "59f980bd-b9ee-44be-ba9c-aeb829a96598"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:46:00.107562 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.107486 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "59f980bd-b9ee-44be-ba9c-aeb829a96598" (UID: "59f980bd-b9ee-44be-ba9c-aeb829a96598"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:46:00.206319 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.206287 2553 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-offline-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:46:00.206319 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.206316 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm9m6\" (UniqueName: \"kubernetes.io/projected/59f980bd-b9ee-44be-ba9c-aeb829a96598-kube-api-access-nm9m6\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:46:00.206319 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.206327 2553 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-ui-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:46:00.206488 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.206335 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:46:00.206488 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.206344 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59f980bd-b9ee-44be-ba9c-aeb829a96598-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:46:00.206488 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.206352 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59f980bd-b9ee-44be-ba9c-aeb829a96598-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:46:00.972923 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.972890 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" event={"ID":"59f980bd-b9ee-44be-ba9c-aeb829a96598","Type":"ContainerDied","Data":"d4c9ec93f9465599b2cdc3a61e55262fd9cd322c8833be6b716cfdaba8618153"} Sep 09 17:46:00.973373 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.972950 2553 scope.go:117] "RemoveContainer" containerID="b14603af5fe81ce0ffeac1d908401209ae150058d647b9e78eaa5d6ab1d7fae2" Sep 09 17:46:00.973373 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.972906 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk" Sep 09 17:46:00.981222 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.981198 2553 scope.go:117] "RemoveContainer" containerID="f4cee327ad38809dc57b1d19a7fa1b95966bdf16a3f82849f702bddce261edff" Sep 09 17:46:00.988450 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.988432 2553 scope.go:117] "RemoveContainer" containerID="9a939723fb527681a097f14976d06fec57705f59a58c6789222fc7c796a1e67d" Sep 09 17:46:00.994819 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.994796 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk"] Sep 09 17:46:00.998293 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.998272 2553 scope.go:117] "RemoveContainer" containerID="c766f8d4749d66a1314ee4a38e266d26bfd0a930789f77a076f829190cdb06e2" Sep 09 17:46:00.998860 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:00.998841 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l99pk"] Sep 09 17:46:01.005524 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:01.005510 2553 scope.go:117] "RemoveContainer" containerID="e75d15022dcdb76bddbddf26b0e0996ae03ffba61e045ea6ee4a2d96e91b9109" Sep 09 17:46:01.012319 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:01.012300 2553 scope.go:117] "RemoveContainer" containerID="ef72f96de4bbadb2f1773d6a469a75b1a5b346a1b29fd1928d6ba8f9891fd7cd" Sep 09 17:46:02.346320 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:02.346276 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" path="/var/lib/kubelet/pods/59f980bd-b9ee-44be-ba9c-aeb829a96598/volumes" Sep 09 17:46:54.730022 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.729983 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp"] Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730266 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="registry" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730276 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="registry" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730316 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="offline" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730322 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="offline" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730331 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="feast-init" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730336 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="feast-init" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730343 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="online" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730349 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="online" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730359 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="ui" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730367 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="ui" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730378 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="feast-apply" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730384 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="feast-apply" Sep 09 17:46:54.730439 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730442 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="online" Sep 09 17:46:54.730870 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730454 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="offline" Sep 09 17:46:54.730870 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730459 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="ui" Sep 09 17:46:54.730870 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.730467 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f980bd-b9ee-44be-ba9c-aeb829a96598" containerName="registry" Sep 09 17:46:54.733494 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.733478 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.736204 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.736177 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 17:46:54.736204 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.736193 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 17:46:54.736487 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.736202 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-t7jfr\"" Sep 09 17:46:54.736487 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.736256 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 17:46:54.736487 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.736196 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 17:46:54.736658 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.736575 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 17:46:54.737556 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.737536 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 17:46:54.744079 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.744057 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp"] Sep 09 17:46:54.822988 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.822950 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.823172 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.822996 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/764daae0-e1bb-4bce-80f1-fb54df508452-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.823172 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.823025 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/764daae0-e1bb-4bce-80f1-fb54df508452-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.823172 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.823050 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkc6f\" (UniqueName: \"kubernetes.io/projected/764daae0-e1bb-4bce-80f1-fb54df508452-kube-api-access-nkc6f\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.823172 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.823084 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.823172 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.823112 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.924021 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.923971 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.924021 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.924029 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.924249 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.924068 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.924249 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.924097 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/764daae0-e1bb-4bce-80f1-fb54df508452-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.924249 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:46:54.924125 2553 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 17:46:54.924249 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:46:54.924212 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls podName:764daae0-e1bb-4bce-80f1-fb54df508452 nodeName:}" failed. No retries permitted until 2026-09-09 17:46:55.42417932 +0000 UTC m=+1689.664085664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" (UID: "764daae0-e1bb-4bce-80f1-fb54df508452") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 17:46:54.924249 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.924236 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/764daae0-e1bb-4bce-80f1-fb54df508452-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.924523 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.924259 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkc6f\" (UniqueName: \"kubernetes.io/projected/764daae0-e1bb-4bce-80f1-fb54df508452-kube-api-access-nkc6f\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.924609 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.924578 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/764daae0-e1bb-4bce-80f1-fb54df508452-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.924963 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.924941 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/764daae0-e1bb-4bce-80f1-fb54df508452-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.926519 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.926495 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.926658 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.926638 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:54.933351 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:54.933330 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkc6f\" (UniqueName: \"kubernetes.io/projected/764daae0-e1bb-4bce-80f1-fb54df508452-kube-api-access-nkc6f\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:55.429143 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:55.429113 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:55.429303 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:46:55.429217 2553 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 17:46:55.429303 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:46:55.429276 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls podName:764daae0-e1bb-4bce-80f1-fb54df508452 nodeName:}" failed. No retries permitted until 2026-09-09 17:46:56.429262123 +0000 UTC m=+1690.669168453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" (UID: "764daae0-e1bb-4bce-80f1-fb54df508452") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 17:46:56.437066 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:56.437033 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:56.439368 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:56.439344 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-s5jjp\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:56.542446 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:56.542415 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:46:56.818917 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:56.818894 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp"] Sep 09 17:46:56.820354 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:46:56.820324 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod764daae0_e1bb_4bce_80f1_fb54df508452.slice/crio-ae16da8c1196ea716949f8c3285f6b54f2b42e197c77e8b1b4f0b75311721190 WatchSource:0}: Error finding container ae16da8c1196ea716949f8c3285f6b54f2b42e197c77e8b1b4f0b75311721190: Status 404 returned error can't find the container with id ae16da8c1196ea716949f8c3285f6b54f2b42e197c77e8b1b4f0b75311721190 Sep 09 17:46:57.188100 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:57.188064 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerStarted","Data":"0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5"} Sep 09 17:46:57.188295 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:46:57.188110 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerStarted","Data":"ae16da8c1196ea716949f8c3285f6b54f2b42e197c77e8b1b4f0b75311721190"} Sep 09 17:47:02.208996 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:02.208961 2553 generic.go:356] "Generic (PLEG): container finished" podID="764daae0-e1bb-4bce-80f1-fb54df508452" containerID="0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5" exitCode=0 Sep 09 17:47:02.209386 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:02.209010 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerDied","Data":"0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5"} Sep 09 17:47:03.214930 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:03.214896 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerStarted","Data":"a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb"} Sep 09 17:47:24.296787 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:24.296751 2553 generic.go:356] "Generic (PLEG): container finished" podID="764daae0-e1bb-4bce-80f1-fb54df508452" containerID="a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb" exitCode=0 Sep 09 17:47:24.297191 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:24.296805 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerDied","Data":"a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb"} Sep 09 17:47:25.305330 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:25.305290 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerStarted","Data":"4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38"} Sep 09 17:47:25.305851 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:25.305341 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerStarted","Data":"dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1"} Sep 09 17:47:25.305851 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:25.305354 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerStarted","Data":"c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91"} Sep 09 17:47:25.335838 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:25.335749 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podStartSLOduration=31.335725008 podStartE2EDuration="31.335725008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:47:25.333499993 +0000 UTC m=+1719.573406344" watchObservedRunningTime="2026-09-09 17:47:25.335725008 +0000 UTC m=+1719.575631363" Sep 09 17:47:26.543283 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:26.543238 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:26.543756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:26.543308 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:26.543756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:26.543329 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:26.543756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:26.543346 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:26.543756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:26.543362 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:26.543756 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:26.543461 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:26.545577 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:26.545535 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.33:8816: connect: connection refused" Sep 09 17:47:26.545747 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:26.545568 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Sep 09 17:47:26.545747 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:26.545584 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 09 17:47:29.543938 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:29.543875 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Sep 09 17:47:29.543938 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:29.543906 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 09 17:47:29.544502 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:29.543878 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.33:8816: connect: connection refused" Sep 09 17:47:32.542917 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:32.542867 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Sep 09 17:47:32.542917 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:32.542891 2553 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 09 17:47:32.543612 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:32.543503 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:32.545789 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:32.545769 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:35.543914 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:35.543877 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:35.544707 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:35.544687 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:36.255127 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:36.255095 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:37.067192 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:37.067153 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:47:37.546807 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:37.546742 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 17:47:57.547749 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:47:57.547711 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 17:48:17.546767 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:17.546718 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 17:48:37.547784 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:37.547731 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 17:48:38.984490 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:38.984401 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp"] Sep 09 17:48:38.984932 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:38.984645 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" containerID="cri-o://c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91" gracePeriod=30 Sep 09 17:48:38.984932 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:38.984728 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="offline" containerID="cri-o://dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1" gracePeriod=30 Sep 09 17:48:38.984932 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:38.984749 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="ui" containerID="cri-o://4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38" gracePeriod=30 Sep 09 17:48:39.155656 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:39.155598 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf"] Sep 09 17:48:39.156192 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:39.155903 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="registry" containerID="cri-o://614d8d2030d1663e785e4ec2310486d4d35330d420faa205d3a34e96db4ba302" gracePeriod=30 Sep 09 17:48:39.156192 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:39.155966 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="offline" containerID="cri-o://db28c97f7a9c12fca0c9abddcded3883c5e0708fd42bd6108e638e31ed3c06eb" gracePeriod=30 Sep 09 17:48:39.156192 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:39.155995 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="online" containerID="cri-o://199f37dfcbdc457f1de14ba8e07cd9e5ef6d249dbd48b2606b1f772b35352f20" gracePeriod=30 Sep 09 17:48:39.156192 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:39.156071 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="ui" containerID="cri-o://a0728036db57c2ca5ab240d392b162ef7b0c5127dbf603ef061aeda7248cf929" gracePeriod=30 Sep 09 17:48:40.611909 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:40.611871 2553 generic.go:356] "Generic (PLEG): container finished" podID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerID="a0728036db57c2ca5ab240d392b162ef7b0c5127dbf603ef061aeda7248cf929" exitCode=0 Sep 09 17:48:40.611909 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:40.611900 2553 generic.go:356] "Generic (PLEG): container finished" podID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerID="199f37dfcbdc457f1de14ba8e07cd9e5ef6d249dbd48b2606b1f772b35352f20" exitCode=0 Sep 09 17:48:40.612330 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:40.611941 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerDied","Data":"a0728036db57c2ca5ab240d392b162ef7b0c5127dbf603ef061aeda7248cf929"} Sep 09 17:48:40.612330 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:40.611983 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerDied","Data":"199f37dfcbdc457f1de14ba8e07cd9e5ef6d249dbd48b2606b1f772b35352f20"} Sep 09 17:48:40.614465 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:40.614441 2553 generic.go:356] "Generic (PLEG): container finished" podID="764daae0-e1bb-4bce-80f1-fb54df508452" containerID="4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38" exitCode=0 Sep 09 17:48:40.614465 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:40.614462 2553 generic.go:356] "Generic (PLEG): container finished" podID="764daae0-e1bb-4bce-80f1-fb54df508452" containerID="c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91" exitCode=0 Sep 09 17:48:40.614646 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:40.614513 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerDied","Data":"4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38"} Sep 09 17:48:40.614646 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:40.614545 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerDied","Data":"c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91"} Sep 09 17:48:41.930950 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:41.930907 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 09 17:48:41.931389 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:41.930943 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 17:48:45.545082 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:45.545040 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Sep 09 17:48:46.361034 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:46.360985 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 09 17:48:51.930955 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:51.930910 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 17:48:51.930955 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:51.930955 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 09 17:48:55.545079 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:55.545032 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Sep 09 17:48:56.361419 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:48:56.361381 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 09 17:49:01.931561 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:01.931503 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 09 17:49:01.932042 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:01.931517 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 17:49:01.932042 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:01.931697 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:49:01.932042 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:01.931769 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:49:05.544686 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:05.544614 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Sep 09 17:49:05.545113 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:05.544797 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:49:06.361208 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:06.361167 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Sep 09 17:49:06.361388 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:06.361275 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:49:09.627648 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.627607 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:49:09.644612 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.644587 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkc6f\" (UniqueName: \"kubernetes.io/projected/764daae0-e1bb-4bce-80f1-fb54df508452-kube-api-access-nkc6f\") pod \"764daae0-e1bb-4bce-80f1-fb54df508452\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " Sep 09 17:49:09.644724 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.644648 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-online-tls\") pod \"764daae0-e1bb-4bce-80f1-fb54df508452\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " Sep 09 17:49:09.644765 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.644721 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/764daae0-e1bb-4bce-80f1-fb54df508452-registry-tls\") pod \"764daae0-e1bb-4bce-80f1-fb54df508452\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " Sep 09 17:49:09.644806 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.644763 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-offline-tls\") pod \"764daae0-e1bb-4bce-80f1-fb54df508452\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " Sep 09 17:49:09.644806 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.644801 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls\") pod \"764daae0-e1bb-4bce-80f1-fb54df508452\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " Sep 09 17:49:09.644887 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.644846 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/764daae0-e1bb-4bce-80f1-fb54df508452-feast-data\") pod \"764daae0-e1bb-4bce-80f1-fb54df508452\" (UID: \"764daae0-e1bb-4bce-80f1-fb54df508452\") " Sep 09 17:49:09.645105 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.645078 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764daae0-e1bb-4bce-80f1-fb54df508452-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "764daae0-e1bb-4bce-80f1-fb54df508452" (UID: "764daae0-e1bb-4bce-80f1-fb54df508452"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:49:09.645497 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.645458 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764daae0-e1bb-4bce-80f1-fb54df508452-feast-data" (OuterVolumeSpecName: "feast-data") pod "764daae0-e1bb-4bce-80f1-fb54df508452" (UID: "764daae0-e1bb-4bce-80f1-fb54df508452"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:49:09.646923 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.646898 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764daae0-e1bb-4bce-80f1-fb54df508452-kube-api-access-nkc6f" (OuterVolumeSpecName: "kube-api-access-nkc6f") pod "764daae0-e1bb-4bce-80f1-fb54df508452" (UID: "764daae0-e1bb-4bce-80f1-fb54df508452"). InnerVolumeSpecName "kube-api-access-nkc6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:49:09.647006 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.646943 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-online-tls" (OuterVolumeSpecName: "online-tls") pod "764daae0-e1bb-4bce-80f1-fb54df508452" (UID: "764daae0-e1bb-4bce-80f1-fb54df508452"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:49:09.647176 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.647160 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "764daae0-e1bb-4bce-80f1-fb54df508452" (UID: "764daae0-e1bb-4bce-80f1-fb54df508452"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:49:09.647214 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.647174 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "764daae0-e1bb-4bce-80f1-fb54df508452" (UID: "764daae0-e1bb-4bce-80f1-fb54df508452"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:49:09.728709 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.728679 2553 generic.go:356] "Generic (PLEG): container finished" podID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerID="db28c97f7a9c12fca0c9abddcded3883c5e0708fd42bd6108e638e31ed3c06eb" exitCode=137 Sep 09 17:49:09.728709 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.728706 2553 generic.go:356] "Generic (PLEG): container finished" podID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerID="614d8d2030d1663e785e4ec2310486d4d35330d420faa205d3a34e96db4ba302" exitCode=137 Sep 09 17:49:09.728887 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.728755 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerDied","Data":"db28c97f7a9c12fca0c9abddcded3883c5e0708fd42bd6108e638e31ed3c06eb"} Sep 09 17:49:09.728887 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.728806 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerDied","Data":"614d8d2030d1663e785e4ec2310486d4d35330d420faa205d3a34e96db4ba302"} Sep 09 17:49:09.731516 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.731496 2553 generic.go:356] "Generic (PLEG): container finished" podID="764daae0-e1bb-4bce-80f1-fb54df508452" containerID="dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1" exitCode=137 Sep 09 17:49:09.731651 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.731568 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" Sep 09 17:49:09.731651 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.731580 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerDied","Data":"dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1"} Sep 09 17:49:09.731651 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.731636 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp" event={"ID":"764daae0-e1bb-4bce-80f1-fb54df508452","Type":"ContainerDied","Data":"ae16da8c1196ea716949f8c3285f6b54f2b42e197c77e8b1b4f0b75311721190"} Sep 09 17:49:09.731774 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.731665 2553 scope.go:117] "RemoveContainer" containerID="4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38" Sep 09 17:49:09.741840 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.741820 2553 scope.go:117] "RemoveContainer" containerID="dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1" Sep 09 17:49:09.745754 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.745736 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/764daae0-e1bb-4bce-80f1-fb54df508452-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:09.745911 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.745759 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkc6f\" (UniqueName: \"kubernetes.io/projected/764daae0-e1bb-4bce-80f1-fb54df508452-kube-api-access-nkc6f\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:09.745911 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.745772 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:09.745911 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.745784 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/764daae0-e1bb-4bce-80f1-fb54df508452-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:09.745911 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.745796 2553 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-offline-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:09.745911 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.745807 2553 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/764daae0-e1bb-4bce-80f1-fb54df508452-ui-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:09.750101 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.750085 2553 scope.go:117] "RemoveContainer" containerID="c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91" Sep 09 17:49:09.755353 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.755304 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp"] Sep 09 17:49:09.759932 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.759912 2553 scope.go:117] "RemoveContainer" containerID="a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb" Sep 09 17:49:09.760952 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.760865 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-s5jjp"] Sep 09 17:49:09.767070 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.767042 2553 scope.go:117] "RemoveContainer" containerID="0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5" Sep 09 17:49:09.774043 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.774023 2553 scope.go:117] "RemoveContainer" containerID="4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38" Sep 09 17:49:09.774324 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:49:09.774300 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38\": container with ID starting with 4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38 not found: ID does not exist" containerID="4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38" Sep 09 17:49:09.774407 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.774333 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38"} err="failed to get container status \"4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38\": rpc error: code = NotFound desc = could not find container \"4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38\": container with ID starting with 4dfa362ae84c2fb84d99cf7ee9681f946a2cb81d933395ed59a003a65d756a38 not found: ID does not exist" Sep 09 17:49:09.774407 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.774352 2553 scope.go:117] "RemoveContainer" containerID="dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1" Sep 09 17:49:09.774615 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:49:09.774598 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1\": container with ID starting with dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1 not found: ID does not exist" containerID="dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1" Sep 09 17:49:09.774688 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.774631 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1"} err="failed to get container status \"dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1\": rpc error: code = NotFound desc = could not find container \"dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1\": container with ID starting with dc57880ffc9efa7126a0c3d81273f23be1b640949746f781fde5140467cc90f1 not found: ID does not exist" Sep 09 17:49:09.774688 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.774645 2553 scope.go:117] "RemoveContainer" containerID="c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91" Sep 09 17:49:09.774861 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:49:09.774846 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91\": container with ID starting with c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91 not found: ID does not exist" containerID="c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91" Sep 09 17:49:09.774903 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.774865 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91"} err="failed to get container status \"c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91\": rpc error: code = NotFound desc = could not find container \"c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91\": container with ID starting with c02f428a269fbacb7a038d0b540660c345e6952e6e138cf8c56f53cdccd17f91 not found: ID does not exist" Sep 09 17:49:09.774903 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.774878 2553 scope.go:117] "RemoveContainer" containerID="a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb" Sep 09 17:49:09.775095 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:49:09.775081 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb\": container with ID starting with a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb not found: ID does not exist" containerID="a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb" Sep 09 17:49:09.775137 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.775099 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb"} err="failed to get container status \"a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb\": rpc error: code = NotFound desc = could not find container \"a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb\": container with ID starting with a481caef166bebf40e5a8e1c601f2f9e0cb45e89da1e06393016b9ce282497cb not found: ID does not exist" Sep 09 17:49:09.775137 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.775111 2553 scope.go:117] "RemoveContainer" containerID="0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5" Sep 09 17:49:09.775326 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:49:09.775305 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5\": container with ID starting with 0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5 not found: ID does not exist" containerID="0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5" Sep 09 17:49:09.775365 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.775333 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5"} err="failed to get container status \"0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5\": rpc error: code = NotFound desc = could not find container \"0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5\": container with ID starting with 0026f2623e021b284b1cadd609ada31b533c5d7383388f9bd7472d7a1004c7e5 not found: ID does not exist" Sep 09 17:49:09.796348 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.796328 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:49:09.846964 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.846940 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-online-tls\") pod \"d806f0c6-ed63-4345-87e1-4ee6b3742339\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " Sep 09 17:49:09.846964 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.846967 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-ui-tls\") pod \"d806f0c6-ed63-4345-87e1-4ee6b3742339\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " Sep 09 17:49:09.847178 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.846994 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-offline-tls\") pod \"d806f0c6-ed63-4345-87e1-4ee6b3742339\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " Sep 09 17:49:09.847178 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.847025 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctr7h\" (UniqueName: \"kubernetes.io/projected/d806f0c6-ed63-4345-87e1-4ee6b3742339-kube-api-access-ctr7h\") pod \"d806f0c6-ed63-4345-87e1-4ee6b3742339\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " Sep 09 17:49:09.847178 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.847154 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-registry-tls\") pod \"d806f0c6-ed63-4345-87e1-4ee6b3742339\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " Sep 09 17:49:09.847350 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.847205 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d806f0c6-ed63-4345-87e1-4ee6b3742339-feast-data\") pod \"d806f0c6-ed63-4345-87e1-4ee6b3742339\" (UID: \"d806f0c6-ed63-4345-87e1-4ee6b3742339\") " Sep 09 17:49:09.847856 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.847826 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d806f0c6-ed63-4345-87e1-4ee6b3742339-feast-data" (OuterVolumeSpecName: "feast-data") pod "d806f0c6-ed63-4345-87e1-4ee6b3742339" (UID: "d806f0c6-ed63-4345-87e1-4ee6b3742339"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:49:09.849050 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.849026 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d806f0c6-ed63-4345-87e1-4ee6b3742339" (UID: "d806f0c6-ed63-4345-87e1-4ee6b3742339"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:49:09.849131 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.849098 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d806f0c6-ed63-4345-87e1-4ee6b3742339-kube-api-access-ctr7h" (OuterVolumeSpecName: "kube-api-access-ctr7h") pod "d806f0c6-ed63-4345-87e1-4ee6b3742339" (UID: "d806f0c6-ed63-4345-87e1-4ee6b3742339"). InnerVolumeSpecName "kube-api-access-ctr7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:49:09.849185 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.849167 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "d806f0c6-ed63-4345-87e1-4ee6b3742339" (UID: "d806f0c6-ed63-4345-87e1-4ee6b3742339"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:49:09.849445 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.849431 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-online-tls" (OuterVolumeSpecName: "online-tls") pod "d806f0c6-ed63-4345-87e1-4ee6b3742339" (UID: "d806f0c6-ed63-4345-87e1-4ee6b3742339"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:49:09.849484 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.849451 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d806f0c6-ed63-4345-87e1-4ee6b3742339" (UID: "d806f0c6-ed63-4345-87e1-4ee6b3742339"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:49:09.948121 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.948083 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-registry-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:09.948121 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.948110 2553 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d806f0c6-ed63-4345-87e1-4ee6b3742339-feast-data\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:09.948121 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.948119 2553 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-online-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:09.948121 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.948127 2553 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-ui-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:09.948367 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.948134 2553 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d806f0c6-ed63-4345-87e1-4ee6b3742339-offline-tls\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:09.948367 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:09.948142 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctr7h\" (UniqueName: \"kubernetes.io/projected/d806f0c6-ed63-4345-87e1-4ee6b3742339-kube-api-access-ctr7h\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:10.351833 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:10.351794 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" path="/var/lib/kubelet/pods/764daae0-e1bb-4bce-80f1-fb54df508452/volumes" Sep 09 17:49:10.741443 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:10.741409 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" event={"ID":"d806f0c6-ed63-4345-87e1-4ee6b3742339","Type":"ContainerDied","Data":"6310e2aaa15e4625bdb00d4d36f4ab6940839a7ab48a405cb57537bc221ab12a"} Sep 09 17:49:10.741903 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:10.741467 2553 scope.go:117] "RemoveContainer" containerID="a0728036db57c2ca5ab240d392b162ef7b0c5127dbf603ef061aeda7248cf929" Sep 09 17:49:10.741903 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:10.741425 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf" Sep 09 17:49:10.749272 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:10.749251 2553 scope.go:117] "RemoveContainer" containerID="db28c97f7a9c12fca0c9abddcded3883c5e0708fd42bd6108e638e31ed3c06eb" Sep 09 17:49:10.756654 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:10.756635 2553 scope.go:117] "RemoveContainer" containerID="199f37dfcbdc457f1de14ba8e07cd9e5ef6d249dbd48b2606b1f772b35352f20" Sep 09 17:49:10.761895 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:10.761873 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf"] Sep 09 17:49:10.764247 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:10.764227 2553 scope.go:117] "RemoveContainer" containerID="614d8d2030d1663e785e4ec2310486d4d35330d420faa205d3a34e96db4ba302" Sep 09 17:49:10.766160 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:10.766139 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-wk8jf"] Sep 09 17:49:10.771750 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:10.771734 2553 scope.go:117] "RemoveContainer" containerID="abd24489c750c3d6ba8f62bb7cb20c973e1ee8f888d7f7844b9a968607750ebc" Sep 09 17:49:10.778682 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:10.778662 2553 scope.go:117] "RemoveContainer" containerID="d38a73e8b6bd6044ac42cc2deec5111e4c6d1eca514e8f2bbca082ff8c651ea9" Sep 09 17:49:12.346569 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:12.346539 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" path="/var/lib/kubelet/pods/d806f0c6-ed63-4345-87e1-4ee6b3742339/volumes" Sep 09 17:49:28.584970 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.584937 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6t78f/must-gather-442pt"] Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585207 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="ui" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585219 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="ui" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585228 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="feast-apply" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585234 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="feast-apply" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585245 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="ui" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585250 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="ui" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585258 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="offline" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585263 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="offline" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585269 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="feast-init" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585275 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="feast-init" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585281 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="online" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585287 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="online" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585295 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="feast-apply" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585301 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="feast-apply" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585306 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="offline" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585311 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="offline" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585320 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="feast-init" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585328 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="feast-init" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585339 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585346 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585361 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="registry" Sep 09 17:49:28.585405 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585368 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="registry" Sep 09 17:49:28.586157 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585432 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="online" Sep 09 17:49:28.586157 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585439 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="online" Sep 09 17:49:28.586157 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585445 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="ui" Sep 09 17:49:28.586157 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585450 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="offline" Sep 09 17:49:28.586157 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585455 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="registry" Sep 09 17:49:28.586157 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585462 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="764daae0-e1bb-4bce-80f1-fb54df508452" containerName="offline" Sep 09 17:49:28.586157 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.585467 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d806f0c6-ed63-4345-87e1-4ee6b3742339" containerName="ui" Sep 09 17:49:28.588414 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.588398 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:28.591522 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.591500 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6t78f\"/\"openshift-service-ca.crt\"" Sep 09 17:49:28.591692 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.591504 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6t78f\"/\"default-dockercfg-mnqkk\"" Sep 09 17:49:28.592821 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.592798 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6t78f\"/\"kube-root-ca.crt\"" Sep 09 17:49:28.598233 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.598212 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6t78f/must-gather-442pt"] Sep 09 17:49:28.694111 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.694074 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc765-a956-4e5e-80b0-09d2d3280754-must-gather-output\") pod \"must-gather-442pt\" (UID: \"cfbdc765-a956-4e5e-80b0-09d2d3280754\") " pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:28.694292 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.694137 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmxmt\" (UniqueName: \"kubernetes.io/projected/cfbdc765-a956-4e5e-80b0-09d2d3280754-kube-api-access-lmxmt\") pod \"must-gather-442pt\" (UID: \"cfbdc765-a956-4e5e-80b0-09d2d3280754\") " pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:28.795395 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.795348 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc765-a956-4e5e-80b0-09d2d3280754-must-gather-output\") pod \"must-gather-442pt\" (UID: \"cfbdc765-a956-4e5e-80b0-09d2d3280754\") " pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:28.795588 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.795425 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmxmt\" (UniqueName: \"kubernetes.io/projected/cfbdc765-a956-4e5e-80b0-09d2d3280754-kube-api-access-lmxmt\") pod \"must-gather-442pt\" (UID: \"cfbdc765-a956-4e5e-80b0-09d2d3280754\") " pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:28.795760 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.795736 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc765-a956-4e5e-80b0-09d2d3280754-must-gather-output\") pod \"must-gather-442pt\" (UID: \"cfbdc765-a956-4e5e-80b0-09d2d3280754\") " pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:28.805242 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.805221 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmxmt\" (UniqueName: \"kubernetes.io/projected/cfbdc765-a956-4e5e-80b0-09d2d3280754-kube-api-access-lmxmt\") pod \"must-gather-442pt\" (UID: \"cfbdc765-a956-4e5e-80b0-09d2d3280754\") " pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:28.904799 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:28.904714 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:29.170733 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:29.170634 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6t78f/must-gather-442pt"] Sep 09 17:49:29.173458 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:49:29.173425 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfbdc765_a956_4e5e_80b0_09d2d3280754.slice/crio-5835a9a0bbac660b12bd3a6c1d44caafdd664793f45738f0e88a6bbf3f5b8765 WatchSource:0}: Error finding container 5835a9a0bbac660b12bd3a6c1d44caafdd664793f45738f0e88a6bbf3f5b8765: Status 404 returned error can't find the container with id 5835a9a0bbac660b12bd3a6c1d44caafdd664793f45738f0e88a6bbf3f5b8765 Sep 09 17:49:29.175179 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:29.175161 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 17:49:29.813910 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:29.813871 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t78f/must-gather-442pt" event={"ID":"cfbdc765-a956-4e5e-80b0-09d2d3280754","Type":"ContainerStarted","Data":"5835a9a0bbac660b12bd3a6c1d44caafdd664793f45738f0e88a6bbf3f5b8765"} Sep 09 17:49:35.849528 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:35.849435 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t78f/must-gather-442pt" event={"ID":"cfbdc765-a956-4e5e-80b0-09d2d3280754","Type":"ContainerStarted","Data":"c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281"} Sep 09 17:49:35.849528 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:35.849492 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t78f/must-gather-442pt" event={"ID":"cfbdc765-a956-4e5e-80b0-09d2d3280754","Type":"ContainerStarted","Data":"49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1"} Sep 09 17:49:35.866889 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:35.866836 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6t78f/must-gather-442pt" podStartSLOduration=1.466857669 podStartE2EDuration="7.866819455s" totalImagesPullingTime="6.399961786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:49:35.865905983 +0000 UTC m=+1850.105812336" watchObservedRunningTime="2026-09-09 17:49:35.866819455 +0000 UTC m=+1850.106725805" Sep 09 17:49:45.889929 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:45.889896 2553 generic.go:356] "Generic (PLEG): container finished" podID="cfbdc765-a956-4e5e-80b0-09d2d3280754" containerID="49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1" exitCode=0 Sep 09 17:49:45.890346 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:45.889979 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t78f/must-gather-442pt" event={"ID":"cfbdc765-a956-4e5e-80b0-09d2d3280754","Type":"ContainerDied","Data":"49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1"} Sep 09 17:49:45.890346 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:45.890298 2553 scope.go:117] "RemoveContainer" containerID="49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1" Sep 09 17:49:46.208427 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:46.208400 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6t78f_must-gather-442pt_cfbdc765-a956-4e5e-80b0-09d2d3280754/gather/0.log" Sep 09 17:49:49.540229 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:49.540199 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zlhb4_22945efa-5c74-4bb9-b708-41a486a7e7c8/global-pull-secret-syncer/0.log" Sep 09 17:49:49.618980 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:49.618944 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q7ctz_095893a3-d652-49a4-b6ae-aa6e67db5938/konnectivity-agent/0.log" Sep 09 17:49:49.722194 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:49.722162 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-213.ec2.internal_0a288a822be8e393a56ffcbb08fdf962/haproxy/0.log" Sep 09 17:49:51.544792 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.544758 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-6t78f/must-gather-442pt"] Sep 09 17:49:51.545247 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.544957 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-6t78f/must-gather-442pt" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" containerName="copy" containerID="cri-o://c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281" gracePeriod=2 Sep 09 17:49:51.547525 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:49:51.547487 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-442pt\" is forbidden: User \"system:node:ip-10-0-141-213.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6t78f\": no relationship found between node 'ip-10-0-141-213.ec2.internal' and this object" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:51.548569 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.548540 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-6t78f/must-gather-442pt"] Sep 09 17:49:51.768820 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.768798 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6t78f_must-gather-442pt_cfbdc765-a956-4e5e-80b0-09d2d3280754/copy/0.log" Sep 09 17:49:51.770248 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.770231 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:51.772851 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:49:51.772828 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-442pt\" is forbidden: User \"system:node:ip-10-0-141-213.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6t78f\": no relationship found between node 'ip-10-0-141-213.ec2.internal' and this object" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:51.889642 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.889557 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc765-a956-4e5e-80b0-09d2d3280754-must-gather-output\") pod \"cfbdc765-a956-4e5e-80b0-09d2d3280754\" (UID: \"cfbdc765-a956-4e5e-80b0-09d2d3280754\") " Sep 09 17:49:51.889642 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.889593 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmxmt\" (UniqueName: \"kubernetes.io/projected/cfbdc765-a956-4e5e-80b0-09d2d3280754-kube-api-access-lmxmt\") pod \"cfbdc765-a956-4e5e-80b0-09d2d3280754\" (UID: \"cfbdc765-a956-4e5e-80b0-09d2d3280754\") " Sep 09 17:49:51.889999 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.889974 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbdc765-a956-4e5e-80b0-09d2d3280754-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "cfbdc765-a956-4e5e-80b0-09d2d3280754" (UID: "cfbdc765-a956-4e5e-80b0-09d2d3280754"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:49:51.891853 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.891830 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbdc765-a956-4e5e-80b0-09d2d3280754-kube-api-access-lmxmt" (OuterVolumeSpecName: "kube-api-access-lmxmt") pod "cfbdc765-a956-4e5e-80b0-09d2d3280754" (UID: "cfbdc765-a956-4e5e-80b0-09d2d3280754"). InnerVolumeSpecName "kube-api-access-lmxmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:49:51.914633 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.914603 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6t78f_must-gather-442pt_cfbdc765-a956-4e5e-80b0-09d2d3280754/copy/0.log" Sep 09 17:49:51.914964 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.914944 2553 generic.go:356] "Generic (PLEG): container finished" podID="cfbdc765-a956-4e5e-80b0-09d2d3280754" containerID="c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281" exitCode=143 Sep 09 17:49:51.915040 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.915000 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:51.915040 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.915015 2553 scope.go:117] "RemoveContainer" containerID="c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281" Sep 09 17:49:51.917381 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:49:51.917355 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-442pt\" is forbidden: User \"system:node:ip-10-0-141-213.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6t78f\": no relationship found between node 'ip-10-0-141-213.ec2.internal' and this object" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:51.922653 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.922633 2553 scope.go:117] "RemoveContainer" containerID="49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1" Sep 09 17:49:51.925037 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:49:51.925016 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-442pt\" is forbidden: User \"system:node:ip-10-0-141-213.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6t78f\": no relationship found between node 'ip-10-0-141-213.ec2.internal' and this object" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" pod="openshift-must-gather-6t78f/must-gather-442pt" Sep 09 17:49:51.934231 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.934215 2553 scope.go:117] "RemoveContainer" containerID="c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281" Sep 09 17:49:51.934486 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:49:51.934471 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281\": container with ID starting with c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281 not found: ID does not exist" containerID="c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281" Sep 09 17:49:51.934531 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.934493 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281"} err="failed to get container status \"c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281\": rpc error: code = NotFound desc = could not find container \"c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281\": container with ID starting with c0c31983e17643b1d3cfca34042ddf665c2b5b96c73802a33b675550a428c281 not found: ID does not exist" Sep 09 17:49:51.934531 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.934508 2553 scope.go:117] "RemoveContainer" containerID="49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1" Sep 09 17:49:51.934746 ip-10-0-141-213 kubenswrapper[2553]: E0909 17:49:51.934727 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1\": container with ID starting with 49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1 not found: ID does not exist" containerID="49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1" Sep 09 17:49:51.934793 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.934754 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1"} err="failed to get container status \"49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1\": rpc error: code = NotFound desc = could not find container \"49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1\": container with ID starting with 49a18a1674a80b51714dacbd40e1fec08da20744676ccc584d9db70bbebc28d1 not found: ID does not exist" Sep 09 17:49:51.990639 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.990594 2553 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cfbdc765-a956-4e5e-80b0-09d2d3280754-must-gather-output\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:51.990639 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:51.990636 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmxmt\" (UniqueName: \"kubernetes.io/projected/cfbdc765-a956-4e5e-80b0-09d2d3280754-kube-api-access-lmxmt\") on node \"ip-10-0-141-213.ec2.internal\" DevicePath \"\"" Sep 09 17:49:52.346309 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:52.346273 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" path="/var/lib/kubelet/pods/cfbdc765-a956-4e5e-80b0-09d2d3280754/volumes" Sep 09 17:49:53.603303 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:53.603271 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e6b1781-a2cb-491a-a6e0-37fbe045bbde/alertmanager/0.log" Sep 09 17:49:53.624683 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:53.624653 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e6b1781-a2cb-491a-a6e0-37fbe045bbde/config-reloader/0.log" Sep 09 17:49:53.643870 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:53.643845 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e6b1781-a2cb-491a-a6e0-37fbe045bbde/kube-rbac-proxy-web/0.log" Sep 09 17:49:53.662779 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:53.662757 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e6b1781-a2cb-491a-a6e0-37fbe045bbde/kube-rbac-proxy/0.log" Sep 09 17:49:53.684389 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:53.684370 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e6b1781-a2cb-491a-a6e0-37fbe045bbde/kube-rbac-proxy-metric/0.log" Sep 09 17:49:53.702160 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:53.702136 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e6b1781-a2cb-491a-a6e0-37fbe045bbde/prom-label-proxy/0.log" Sep 09 17:49:53.721784 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:53.721765 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9e6b1781-a2cb-491a-a6e0-37fbe045bbde/init-config-reloader/0.log" Sep 09 17:49:53.872136 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:53.872038 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84bbb9896d-75tsk_efc30c56-e147-45f4-b69c-5d712eb218b1/metrics-server/0.log" Sep 09 17:49:53.893638 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:53.893589 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-5x4fl_a664f0f0-5cc2-421f-bc9f-d35320a559ab/monitoring-plugin/0.log" Sep 09 17:49:54.071423 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.071392 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x2lqb_c03dc487-852a-4d46-b563-1e2e9c3c68b3/node-exporter/0.log" Sep 09 17:49:54.089211 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.089181 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x2lqb_c03dc487-852a-4d46-b563-1e2e9c3c68b3/kube-rbac-proxy/0.log" Sep 09 17:49:54.109335 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.109309 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x2lqb_c03dc487-852a-4d46-b563-1e2e9c3c68b3/init-textfile/0.log" Sep 09 17:49:54.467174 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.467145 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c4798658f-z8qcd_e4f75adb-8242-4ad5-af1b-41f6df199e27/telemeter-client/0.log" Sep 09 17:49:54.486174 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.486144 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c4798658f-z8qcd_e4f75adb-8242-4ad5-af1b-41f6df199e27/reload/0.log" Sep 09 17:49:54.505328 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.505294 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c4798658f-z8qcd_e4f75adb-8242-4ad5-af1b-41f6df199e27/kube-rbac-proxy/0.log" Sep 09 17:49:54.530077 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.530049 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddf9ddbd-jzljp_abc4748f-3749-4a07-941f-9ba773ab5e8f/thanos-query/0.log" Sep 09 17:49:54.552178 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.552149 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddf9ddbd-jzljp_abc4748f-3749-4a07-941f-9ba773ab5e8f/kube-rbac-proxy-web/0.log" Sep 09 17:49:54.570296 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.570274 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddf9ddbd-jzljp_abc4748f-3749-4a07-941f-9ba773ab5e8f/kube-rbac-proxy/0.log" Sep 09 17:49:54.591888 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.591861 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddf9ddbd-jzljp_abc4748f-3749-4a07-941f-9ba773ab5e8f/prom-label-proxy/0.log" Sep 09 17:49:54.614807 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.614783 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddf9ddbd-jzljp_abc4748f-3749-4a07-941f-9ba773ab5e8f/kube-rbac-proxy-rules/0.log" Sep 09 17:49:54.641307 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:54.641289 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8ddf9ddbd-jzljp_abc4748f-3749-4a07-941f-9ba773ab5e8f/kube-rbac-proxy-metrics/0.log" Sep 09 17:49:56.936751 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.936721 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6"] Sep 09 17:49:56.937144 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.937032 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" containerName="gather" Sep 09 17:49:56.937144 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.937044 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" containerName="gather" Sep 09 17:49:56.937144 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.937054 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" containerName="copy" Sep 09 17:49:56.937144 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.937059 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" containerName="copy" Sep 09 17:49:56.937144 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.937104 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" containerName="copy" Sep 09 17:49:56.937144 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.937114 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfbdc765-a956-4e5e-80b0-09d2d3280754" containerName="gather" Sep 09 17:49:56.938870 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.938848 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:56.941658 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.941639 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k9g2c\"/\"kube-root-ca.crt\"" Sep 09 17:49:56.941761 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.941692 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k9g2c\"/\"openshift-service-ca.crt\"" Sep 09 17:49:56.941880 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.941865 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k9g2c\"/\"default-dockercfg-xblg7\"" Sep 09 17:49:56.955316 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:56.955292 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6"] Sep 09 17:49:57.031954 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.031917 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-podres\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.032119 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.031969 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-lib-modules\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.032119 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.032105 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-sys\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.032249 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.032138 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-proc\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.032249 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.032235 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zcg\" (UniqueName: \"kubernetes.io/projected/f7b40022-b183-405a-98cf-189a0e60ca63-kube-api-access-v9zcg\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.132692 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.132654 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zcg\" (UniqueName: \"kubernetes.io/projected/f7b40022-b183-405a-98cf-189a0e60ca63-kube-api-access-v9zcg\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.132890 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.132706 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-podres\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.132890 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.132733 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-lib-modules\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.132890 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.132787 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-sys\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.132890 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.132813 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-proc\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.133107 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.132884 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-sys\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.133107 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.132892 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-podres\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.133107 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.132913 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-proc\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.133107 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.132912 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7b40022-b183-405a-98cf-189a0e60ca63-lib-modules\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.141211 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.141185 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zcg\" (UniqueName: \"kubernetes.io/projected/f7b40022-b183-405a-98cf-189a0e60ca63-kube-api-access-v9zcg\") pod \"perf-node-gather-daemonset-jbgm6\" (UID: \"f7b40022-b183-405a-98cf-189a0e60ca63\") " pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.247976 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.247887 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.513961 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.513887 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6"] Sep 09 17:49:57.519330 ip-10-0-141-213 kubenswrapper[2553]: W0909 17:49:57.519288 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf7b40022_b183_405a_98cf_189a0e60ca63.slice/crio-60ad057cdf7280f86a6df7217c262f2e25d5260273f3b330272830f5f22679f5 WatchSource:0}: Error finding container 60ad057cdf7280f86a6df7217c262f2e25d5260273f3b330272830f5f22679f5: Status 404 returned error can't find the container with id 60ad057cdf7280f86a6df7217c262f2e25d5260273f3b330272830f5f22679f5 Sep 09 17:49:57.938271 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.938240 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" event={"ID":"f7b40022-b183-405a-98cf-189a0e60ca63","Type":"ContainerStarted","Data":"259d37a74f23d202efd5746cb3884d427e2f27c62f8b06e32545be711755c0aa"} Sep 09 17:49:57.938640 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.938291 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" event={"ID":"f7b40022-b183-405a-98cf-189a0e60ca63","Type":"ContainerStarted","Data":"60ad057cdf7280f86a6df7217c262f2e25d5260273f3b330272830f5f22679f5"} Sep 09 17:49:57.938640 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.938325 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:49:57.955634 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:57.955574 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" podStartSLOduration=1.955551817 podStartE2EDuration="1.955551817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:49:57.954646757 +0000 UTC m=+1872.194553114" watchObservedRunningTime="2026-09-09 17:49:57.955551817 +0000 UTC m=+1872.195458169" Sep 09 17:49:58.300252 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:58.300175 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n6pqz_2ef63ca1-b605-4e44-a748-1319401e5e75/dns/0.log" Sep 09 17:49:58.320870 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:58.320840 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n6pqz_2ef63ca1-b605-4e44-a748-1319401e5e75/kube-rbac-proxy/0.log" Sep 09 17:49:58.343607 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:58.343579 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-54x4s_b88c2732-169f-47a7-b117-b556624997e7/dns-node-resolver/0.log" Sep 09 17:49:58.778740 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:58.778714 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7894976f45-b8sps_e88c93fd-9a23-4b50-96f9-288af4d94959/registry/0.log" Sep 09 17:49:58.840677 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:58.840644 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tt8b9_2a5ff466-6997-438f-8364-7277a690051d/node-ca/0.log" Sep 09 17:49:59.856042 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:49:59.855992 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n7mpt_292acb04-43b0-4a43-a8c4-5ae8da4cae31/serve-healthcheck-canary/0.log" Sep 09 17:50:00.196763 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:00.196735 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2vdjr_4c3f11a8-d34e-4c31-818d-5bd2d17fe053/kube-rbac-proxy/0.log" Sep 09 17:50:00.214778 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:00.214750 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2vdjr_4c3f11a8-d34e-4c31-818d-5bd2d17fe053/exporter/0.log" Sep 09 17:50:00.236588 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:00.236560 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2vdjr_4c3f11a8-d34e-4c31-818d-5bd2d17fe053/extractor/0.log" Sep 09 17:50:03.951946 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:03.951918 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k9g2c/perf-node-gather-daemonset-jbgm6" Sep 09 17:50:05.885052 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:05.884975 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdsct_fc030ff5-217f-41e5-8b48-bc45036de749/kube-multus-additional-cni-plugins/0.log" Sep 09 17:50:05.907014 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:05.906982 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdsct_fc030ff5-217f-41e5-8b48-bc45036de749/egress-router-binary-copy/0.log" Sep 09 17:50:05.926695 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:05.926668 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdsct_fc030ff5-217f-41e5-8b48-bc45036de749/cni-plugins/0.log" Sep 09 17:50:05.947961 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:05.947939 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdsct_fc030ff5-217f-41e5-8b48-bc45036de749/bond-cni-plugin/0.log" Sep 09 17:50:05.971252 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:05.971228 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdsct_fc030ff5-217f-41e5-8b48-bc45036de749/routeoverride-cni/0.log" Sep 09 17:50:05.991462 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:05.991434 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdsct_fc030ff5-217f-41e5-8b48-bc45036de749/whereabouts-cni-bincopy/0.log" Sep 09 17:50:06.010852 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:06.010821 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdsct_fc030ff5-217f-41e5-8b48-bc45036de749/whereabouts-cni/0.log" Sep 09 17:50:06.083333 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:06.083305 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zg2bq_c137012e-8c0e-4444-8c6b-f001185bfc4e/kube-multus/0.log" Sep 09 17:50:06.214098 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:06.214077 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tc2sl_8a9f88fb-c187-4672-bd8e-1916d5f9ff54/network-metrics-daemon/0.log" Sep 09 17:50:06.233371 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:06.233351 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tc2sl_8a9f88fb-c187-4672-bd8e-1916d5f9ff54/kube-rbac-proxy/0.log" Sep 09 17:50:07.364718 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:07.364684 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkdmr_690ed990-fef1-46c2-b214-3d2ba5fb78c4/ovn-controller/0.log" Sep 09 17:50:07.382328 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:07.382298 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkdmr_690ed990-fef1-46c2-b214-3d2ba5fb78c4/ovn-acl-logging/0.log" Sep 09 17:50:07.404659 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:07.404617 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkdmr_690ed990-fef1-46c2-b214-3d2ba5fb78c4/kube-rbac-proxy-node/0.log" Sep 09 17:50:07.426510 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:07.426477 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkdmr_690ed990-fef1-46c2-b214-3d2ba5fb78c4/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 17:50:07.445319 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:07.445292 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkdmr_690ed990-fef1-46c2-b214-3d2ba5fb78c4/northd/0.log" Sep 09 17:50:07.463149 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:07.463113 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkdmr_690ed990-fef1-46c2-b214-3d2ba5fb78c4/nbdb/0.log" Sep 09 17:50:07.482492 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:07.482471 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkdmr_690ed990-fef1-46c2-b214-3d2ba5fb78c4/sbdb/0.log" Sep 09 17:50:07.537344 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:07.537315 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkdmr_690ed990-fef1-46c2-b214-3d2ba5fb78c4/ovnkube-controller/0.log" Sep 09 17:50:08.554636 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:08.554605 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-f6l6s_7dc76395-0bd2-439e-b9f9-2c89105cc455/network-check-target-container/0.log" Sep 09 17:50:09.512332 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:09.512303 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xhtvk_b1028b34-ff23-48f2-b853-0b0cdf0fe595/iptables-alerter/0.log" Sep 09 17:50:10.137358 ip-10-0-141-213 kubenswrapper[2553]: I0909 17:50:10.137333 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-rvdl4_16414bac-4ae6-4cb6-bb88-b1fe2906b3c6/tuned/0.log"