Sep 09 12:02:23.108421 ip-10-0-143-139 systemd[1]: Starting Kubernetes Kubelet... Sep 09 12:02:23.506484 ip-10-0-143-139 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 12:02:23.506484 ip-10-0-143-139 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 12:02:23.506484 ip-10-0-143-139 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 12:02:23.506484 ip-10-0-143-139 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 12:02:23.506484 ip-10-0-143-139 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 12:02:23.507436 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.507345 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 12:02:23.510879 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510864 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 12:02:23.510879 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510879 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510883 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510886 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510889 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510892 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510918 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510922 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510925 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510928 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510931 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510935 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510937 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510940 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510945 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 12:02:23.510944 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510949 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510952 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510963 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510967 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510970 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510973 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510976 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510979 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510982 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510985 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510988 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510992 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510994 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.510997 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511000 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511003 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511005 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511008 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511011 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511013 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 12:02:23.511297 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511016 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511019 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511021 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511024 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511026 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511030 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511034 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511036 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511039 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511042 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511045 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511047 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511050 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511052 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511056 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511059 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511062 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511064 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511067 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 12:02:23.511781 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511069 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511072 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511075 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511077 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511080 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511082 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511085 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511088 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511090 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511093 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511095 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511098 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511100 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511103 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511105 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511108 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511110 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511113 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511115 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 12:02:23.512461 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511118 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511121 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511123 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511126 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511128 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511130 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511133 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511136 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511149 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511152 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511155 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511157 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511160 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511163 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511166 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511168 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511171 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511173 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511176 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511180 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 12:02:23.513000 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511182 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 12:02:23.513515 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511185 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 12:02:23.513515 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511188 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 12:02:23.513515 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511190 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 12:02:23.513515 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511193 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 12:02:23.513515 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.511196 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 12:02:23.514624 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514607 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 12:02:23.514624 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514624 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514628 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514631 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514635 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514640 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514645 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514648 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514651 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514654 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514657 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514660 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514663 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514666 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514669 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514672 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514675 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514678 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514681 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514684 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514687 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 12:02:23.514682 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514691 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514694 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514697 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514699 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514702 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514705 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514707 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514710 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514713 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514716 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514720 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514722 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514726 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514730 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514733 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514735 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514738 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514741 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514744 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 12:02:23.515201 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514746 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514749 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514751 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514754 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514756 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514759 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514762 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514764 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514767 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514769 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514772 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514775 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514779 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514782 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514784 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514787 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514790 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514793 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514796 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514798 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 12:02:23.515672 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514802 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514805 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514808 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514811 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514813 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514816 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514820 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514822 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514825 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514828 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514830 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514833 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514835 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514838 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514840 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514843 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514845 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514848 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514851 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 12:02:23.516184 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514853 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514856 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514858 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514861 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514865 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514868 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514871 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514873 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514877 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514880 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514883 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514886 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514888 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514891 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514894 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514896 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514899 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514902 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514904 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.514907 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 12:02:23.516652 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515313 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515320 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515323 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515326 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515329 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515331 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515334 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515337 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515340 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515343 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515346 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515349 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515352 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515355 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515358 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515360 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515366 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515371 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515374 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 12:02:23.517166 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515378 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515381 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515384 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515387 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515389 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515392 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515395 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515397 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515400 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515403 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515405 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515408 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515410 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515413 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515415 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515418 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515421 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515423 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515426 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 12:02:23.517643 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515428 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515431 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515434 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515436 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515439 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515441 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515445 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515448 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515451 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515453 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515456 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515459 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515462 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515465 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515467 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515470 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515473 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515476 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515479 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515481 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 12:02:23.518126 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515484 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515486 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515489 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515491 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515494 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515496 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515499 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515501 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515504 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515506 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515509 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515511 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515514 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515518 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515522 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515525 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515527 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515530 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515533 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515536 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 12:02:23.518634 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515538 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515541 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515546 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515548 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515552 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515555 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515557 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515561 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515565 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515567 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515570 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515572 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515575 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515577 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515580 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515583 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515585 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515588 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515590 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515593 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 12:02:23.519131 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.515595 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515663 2556 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515701 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515706 2556 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515711 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515715 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515718 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515722 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515726 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515730 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515733 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515737 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515740 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515743 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515746 2556 flags.go:64] FLAG: --cgroup-root="" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515749 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515752 2556 flags.go:64] FLAG: --client-ca-file="" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515756 2556 flags.go:64] FLAG: --cloud-provider="external" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515759 2556 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515763 2556 flags.go:64] FLAG: --cluster-domain="" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515766 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515770 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515773 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515776 2556 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 12:02:23.519751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515784 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515787 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515790 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515793 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515796 2556 flags.go:64] FLAG: --contention-profiling="false" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515799 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515802 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515805 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515808 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515812 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515815 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515818 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515820 2556 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515823 2556 flags.go:64] FLAG: --enable-server="true" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515826 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515830 2556 flags.go:64] FLAG: --event-burst="100" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515835 2556 flags.go:64] FLAG: --event-qps="50" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515837 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515841 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515844 2556 flags.go:64] FLAG: --eviction-hard="" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515848 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515851 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515854 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515857 2556 flags.go:64] FLAG: --eviction-soft="" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515860 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 12:02:23.520349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515862 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515866 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515869 2556 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515872 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515875 2556 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515877 2556 flags.go:64] FLAG: --feature-gates="" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515881 2556 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515884 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515887 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515890 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515893 2556 flags.go:64] FLAG: --healthz-port="10248" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515896 2556 flags.go:64] FLAG: --help="false" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515899 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515902 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515905 2556 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515908 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515911 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515915 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515917 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515920 2556 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515923 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515926 2556 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515929 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515933 2556 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 12:02:23.521012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515936 2556 flags.go:64] FLAG: --kube-reserved="" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515939 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515942 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515945 2556 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515948 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515950 2556 flags.go:64] FLAG: --lock-file="" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515953 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515956 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515959 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515964 2556 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515968 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515971 2556 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515973 2556 flags.go:64] FLAG: --logging-format="text" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515976 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515979 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515982 2556 flags.go:64] FLAG: --manifest-url="" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515985 2556 flags.go:64] FLAG: --manifest-url-header="" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515989 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515992 2556 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515996 2556 flags.go:64] FLAG: --max-pods="110" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.515998 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516001 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516004 2556 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516007 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516010 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 12:02:23.521640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516012 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516015 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516022 2556 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516025 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516028 2556 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516031 2556 flags.go:64] FLAG: --pod-cidr="" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516036 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516041 2556 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516044 2556 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516047 2556 flags.go:64] FLAG: --pods-per-core="0" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516050 2556 flags.go:64] FLAG: --port="10250" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516053 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516056 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02518f4086ac14c73" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516059 2556 flags.go:64] FLAG: --qos-reserved="" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516062 2556 flags.go:64] FLAG: --read-only-port="10255" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516064 2556 flags.go:64] FLAG: --register-node="true" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516067 2556 flags.go:64] FLAG: --register-with-taints="" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516071 2556 flags.go:64] FLAG: --registry-burst="10" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516075 2556 flags.go:64] FLAG: --registry-qps="5" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516078 2556 flags.go:64] FLAG: --reserved-cpus="" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516081 2556 flags.go:64] FLAG: --reserved-memory="" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516084 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516087 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516091 2556 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516094 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 12:02:23.522271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516096 2556 flags.go:64] FLAG: --runonce="false" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516099 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516102 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516105 2556 flags.go:64] FLAG: --seccomp-default="false" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516108 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516110 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516113 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516116 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516119 2556 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516122 2556 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516125 2556 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516128 2556 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516131 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516135 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516153 2556 flags.go:64] FLAG: --system-cgroups="" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516157 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516164 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516167 2556 flags.go:64] FLAG: --tls-cert-file="" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516170 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516175 2556 flags.go:64] FLAG: --tls-min-version="" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516177 2556 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516181 2556 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516183 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516187 2556 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516190 2556 flags.go:64] FLAG: --v="2" Sep 09 12:02:23.522890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516194 2556 flags.go:64] FLAG: --version="false" Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516198 2556 flags.go:64] FLAG: --vmodule="" Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516202 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.516205 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516334 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516338 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516341 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516344 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516348 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516350 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516353 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516355 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516358 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516361 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516364 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516366 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516369 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516371 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516374 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516377 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 12:02:23.523499 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516382 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516387 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516390 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516396 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516399 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516404 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516407 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516410 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516413 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516415 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516418 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516420 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516423 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516426 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516429 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516432 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516435 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516437 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516440 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516442 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 12:02:23.523992 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516445 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516447 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516450 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516453 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516455 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516458 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516460 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516463 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516465 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516468 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516471 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516473 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516477 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516482 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516485 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516489 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516492 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516495 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516498 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 12:02:23.524540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516501 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516504 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516506 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516509 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516511 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516514 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516516 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516519 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516522 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516524 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516527 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516529 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516534 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516537 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516540 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516543 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516546 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516548 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516551 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516554 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 12:02:23.525016 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516556 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516559 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516562 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516564 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516567 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516570 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516573 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516576 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516579 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516582 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516585 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516587 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516589 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516592 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516595 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516597 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516600 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516602 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516605 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516607 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 09 12:02:23.525583 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516610 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516614 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516616 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.516619 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.517397 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.524039 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.524053 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.524860 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.524868 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524921 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524926 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524929 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524933 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524947 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524950 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 12:02:23.526069 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524953 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524957 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524960 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524963 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524966 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524968 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524971 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524974 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524976 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524979 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524982 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524984 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524987 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524990 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524992 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524995 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.524997 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525000 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525002 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 12:02:23.526458 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525005 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525007 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525010 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525012 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525017 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525022 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525026 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525030 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525034 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525037 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525040 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525043 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525046 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525049 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525052 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525055 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525058 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525061 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525064 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 12:02:23.526908 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525067 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525069 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525072 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525075 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525077 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525080 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525083 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525086 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525088 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525091 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525093 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525096 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525099 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525101 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525104 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525107 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525109 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525113 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525116 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525119 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 12:02:23.527392 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525121 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525124 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525127 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525129 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525132 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525134 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525150 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525153 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525156 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525158 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525161 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525163 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525166 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525170 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525173 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525176 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525180 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525183 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525186 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525188 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 12:02:23.527913 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525191 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525194 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525197 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525200 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525203 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525206 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525209 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525212 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525214 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525219 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525223 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525227 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525230 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525234 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525237 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 12:02:23.528540 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.525242 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525351 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525356 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525359 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525362 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525365 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525368 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525371 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525374 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525376 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525379 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525383 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525387 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525390 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525393 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525396 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525399 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525401 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525404 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525407 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 12:02:23.528921 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525409 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525412 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525414 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525417 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525420 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525422 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525425 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525428 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525430 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525434 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525436 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525439 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525441 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525444 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525447 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525449 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525452 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525455 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525457 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 12:02:23.529434 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525460 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525462 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525465 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525468 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525470 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525473 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525476 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525478 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525481 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525483 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525486 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525489 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525491 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525494 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525496 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525499 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525501 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525504 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525507 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525509 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 12:02:23.529901 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525512 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525515 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525518 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525521 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525523 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525527 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525530 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525533 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525537 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525539 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525542 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525544 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525547 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525551 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525555 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525558 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525560 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525563 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 12:02:23.530395 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525566 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525569 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525572 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525574 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525577 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525580 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525582 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525585 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525588 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525591 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525594 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525597 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525599 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525602 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525605 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525607 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525610 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525613 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525616 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525619 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 12:02:23.530840 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525621 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 12:02:23.531362 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525624 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 12:02:23.531362 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:23.525626 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 12:02:23.531362 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.525631 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 12:02:23.531362 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.525768 2556 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 12:02:23.531362 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.527701 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 12:02:23.531362 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.528541 2556 server.go:1019] "Starting client certificate rotation" Sep 09 12:02:23.531362 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.528640 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 12:02:23.531362 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.528677 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 12:02:23.531362 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.530666 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 12:02:23.531871 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.531851 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 12:02:23.541923 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.541905 2556 log.go:25] "Validated CRI v1 runtime API" Sep 09 12:02:23.547363 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.547345 2556 log.go:25] "Validated CRI v1 image API" Sep 09 12:02:23.548211 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.548193 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 12:02:23.552454 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.552436 2556 fs.go:135] Filesystem UUIDs: map[1f8f0bf1-6b00-4e28-81d9-f6781f8d6206:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c0d52a92-ad6a-4802-9645-5e94a6f5b8cf:/dev/nvme0n1p3] Sep 09 12:02:23.552511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.552453 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 12:02:23.557322 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.557218 2556 manager.go:216] Machine: {Timestamp:2026-09-09 12:02:23.556006086 +0000 UTC m=+0.328975150 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100911 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ddcc8332aad06499e7a6db2eecf75 SystemUUID:ec2ddcc8-332a-ad06-499e-7a6db2eecf75 BootID:780eb4dd-27a4-4f52-8070-af272f27e3fb Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7f:2a:7e:00:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7f:2a:7e:00:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:66:67:ff:76:66 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 12:02:23.557322 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.557317 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 12:02:23.557427 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.557408 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 12:02:23.560078 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.560058 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 12:02:23.560222 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.560080 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 12:02:23.560705 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.560689 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 12:02:23.560831 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.560820 2556 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 12:02:23.560871 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.560834 2556 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 12:02:23.560871 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.560842 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 12:02:23.561778 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.561767 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 12:02:23.562601 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.562590 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 12:02:23.565940 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.565929 2556 state_mem.go:36] "Initialized new in-memory state store" Sep 09 12:02:23.566041 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.566032 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 12:02:23.566082 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.566062 2556 kubelet.go:486] "Attempting to sync node with API server" Sep 09 12:02:23.566082 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.566071 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 12:02:23.566082 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.566081 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 12:02:23.566207 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.566091 2556 kubelet.go:398] "Adding apiserver pod source" Sep 09 12:02:23.566207 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.566099 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 12:02:23.566515 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.566504 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 12:02:23.566552 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.566522 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 12:02:23.568591 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.568566 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 12:02:23.571243 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.571228 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 12:02:23.571306 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.571258 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 12:02:23.572353 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572341 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 12:02:23.572423 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572362 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 12:02:23.572423 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572369 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 12:02:23.572423 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572374 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 12:02:23.572423 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572383 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 12:02:23.572423 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572388 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 12:02:23.572423 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572404 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 12:02:23.572423 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572412 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 12:02:23.572423 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572423 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 12:02:23.572640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572429 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 12:02:23.572640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572441 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 12:02:23.572640 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.572447 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 12:02:23.573153 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.573130 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 12:02:23.573207 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.573157 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 12:02:23.576843 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.576827 2556 server.go:1268] "Started kubelet" Sep 09 12:02:23.577037 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.576964 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 12:02:23.577187 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.577108 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 12:02:23.579934 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.579687 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 12:02:23.580840 ip-10-0-143-139 systemd[1]: Started Kubernetes Kubelet. Sep 09 12:02:23.581169 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.580857 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 12:02:23.581975 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.581936 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 12:02:23.581975 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.581931 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 12:02:23.582317 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.582298 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 12:02:23.582510 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.582429 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 12:02:23.584650 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.584636 2556 server.go:310] "Adding debug handlers to kubelet server" Sep 09 12:02:23.587791 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.586934 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-139.ec2.internal.18d3a5681cee945e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-139.ec2.internal,UID:ip-10-0-143-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-139.ec2.internal,},FirstTimestamp:2026-09-09 12:02:23.576798302 +0000 UTC m=+0.349767367,LastTimestamp:2026-09-09 12:02:23.576798302 +0000 UTC m=+0.349767367,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-139.ec2.internal,}" Sep 09 12:02:23.587895 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.587879 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 12:02:23.588395 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.588378 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 12:02:23.588492 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.588473 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:23.589900 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.589882 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 12:02:23.589900 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.589889 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 12:02:23.589900 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.589901 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 12:02:23.590062 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.590004 2556 reconstruct.go:97] "Volume reconstruction finished" Sep 09 12:02:23.590062 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.590010 2556 reconciler.go:29] "Reconciler: start to sync state" Sep 09 12:02:23.592090 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.592074 2556 factory.go:55] Registering systemd factory Sep 09 12:02:23.592170 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.592093 2556 factory.go:223] Registration of the systemd container factory successfully Sep 09 12:02:23.592306 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.592295 2556 factory.go:153] Registering CRI-O factory Sep 09 12:02:23.592306 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.592306 2556 factory.go:223] Registration of the crio container factory successfully Sep 09 12:02:23.592367 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.592352 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 12:02:23.592396 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.592384 2556 factory.go:103] Registering Raw factory Sep 09 12:02:23.592423 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.592400 2556 manager.go:1196] Started watching for new ooms in manager Sep 09 12:02:23.592606 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.592572 2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 12:02:23.592667 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.592611 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 12:02:23.592795 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.592784 2556 manager.go:349] Starting recovery of all containers Sep 09 12:02:23.600836 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.600813 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jrc2q" Sep 09 12:02:23.602721 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.602567 2556 manager.go:354] Recovery completed Sep 09 12:02:23.609057 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.609039 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 12:02:23.610918 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.610897 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory" Sep 09 12:02:23.610987 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.610934 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 12:02:23.610987 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.610945 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID" Sep 09 12:02:23.611526 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.611508 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 12:02:23.611526 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.611519 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 12:02:23.611526 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.611524 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 12:02:23.611701 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.611543 2556 state_mem.go:36] "Initialized new in-memory state store" Sep 09 12:02:23.612193 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.612179 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory" Sep 09 12:02:23.612264 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.612207 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 12:02:23.612264 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.612220 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID" Sep 09 12:02:23.612767 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.612754 2556 policy_none.go:49] "None policy: Start" Sep 09 12:02:23.612806 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.612770 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 12:02:23.612806 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.612779 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 12:02:23.613349 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.613286 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-139.ec2.internal.18d3a5681ef73232 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-139.ec2.internal,UID:ip-10-0-143-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-139.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-139.ec2.internal,},FirstTimestamp:2026-09-09 12:02:23.610917426 +0000 UTC m=+0.383886489,LastTimestamp:2026-09-09 12:02:23.610917426 +0000 UTC m=+0.383886489,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-139.ec2.internal,}" Sep 09 12:02:23.613809 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.613798 2556 policy_none.go:47] "Start" Sep 09 12:02:23.617249 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.617228 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jrc2q" Sep 09 12:02:23.668661 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.668638 2556 manager.go:337] "Starting Device Plugin manager" Sep 09 12:02:23.668783 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.668683 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 12:02:23.668783 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.668696 2556 server.go:87] "Starting device plugin registration server" Sep 09 12:02:23.669063 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.669050 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 12:02:23.669130 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.669066 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 12:02:23.669258 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.669240 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 12:02:23.669357 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.669314 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 12:02:23.669357 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.669322 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 12:02:23.669995 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.669972 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 12:02:23.670077 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.670033 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:23.716113 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.716090 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 12:02:23.717301 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.717284 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 12:02:23.717388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.717332 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 12:02:23.717388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.717353 2556 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 12:02:23.717513 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.717435 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 12:02:23.725364 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.725345 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 12:02:23.769427 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.769361 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 12:02:23.770213 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.770200 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory" Sep 09 12:02:23.770277 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.770227 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 12:02:23.770277 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.770238 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID" Sep 09 12:02:23.770277 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.770259 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.779565 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.779551 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.779624 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.779570 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-139.ec2.internal\": node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:23.816153 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.816121 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:23.818199 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.818182 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"] Sep 09 12:02:23.818268 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.818254 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 12:02:23.819582 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.819568 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory" Sep 09 12:02:23.819669 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.819598 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 12:02:23.819669 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.819611 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID" Sep 09 12:02:23.820875 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.820861 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 12:02:23.820990 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.820976 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.821036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.821008 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 12:02:23.821581 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.821566 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory" Sep 09 12:02:23.821652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.821594 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 12:02:23.821652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.821608 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID" Sep 09 12:02:23.821652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.821611 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory" Sep 09 12:02:23.821652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.821626 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 12:02:23.821652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.821636 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID" Sep 09 12:02:23.822914 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.822899 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.822990 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.822925 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 12:02:23.823763 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.823748 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory" Sep 09 12:02:23.823839 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.823775 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 12:02:23.823839 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.823786 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID" Sep 09 12:02:23.852820 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.852794 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-139.ec2.internal\" not found" node="ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.856192 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.856177 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-139.ec2.internal\" not found" node="ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.892771 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.892755 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/968730223ffcdc713c2b8b2489f9ba2f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-139.ec2.internal\" (UID: \"968730223ffcdc713c2b8b2489f9ba2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.892847 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.892785 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1ca52691c2cdddfad83976f50a01762b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"1ca52691c2cdddfad83976f50a01762b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.892847 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.892803 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ca52691c2cdddfad83976f50a01762b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"1ca52691c2cdddfad83976f50a01762b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.916773 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:23.916759 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:23.993126 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.993089 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ca52691c2cdddfad83976f50a01762b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"1ca52691c2cdddfad83976f50a01762b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.993126 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.993130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/968730223ffcdc713c2b8b2489f9ba2f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-139.ec2.internal\" (UID: \"968730223ffcdc713c2b8b2489f9ba2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.993297 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.993194 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/968730223ffcdc713c2b8b2489f9ba2f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-139.ec2.internal\" (UID: \"968730223ffcdc713c2b8b2489f9ba2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.993297 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.993236 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ca52691c2cdddfad83976f50a01762b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"1ca52691c2cdddfad83976f50a01762b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.993297 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.993257 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1ca52691c2cdddfad83976f50a01762b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"1ca52691c2cdddfad83976f50a01762b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" Sep 09 12:02:23.993297 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:23.993280 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1ca52691c2cdddfad83976f50a01762b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"1ca52691c2cdddfad83976f50a01762b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" Sep 09 12:02:24.017217 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:24.017194 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:24.117985 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:24.117950 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:24.155148 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:24.155119 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" Sep 09 12:02:24.159208 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:24.159189 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" Sep 09 12:02:25.311284 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:25.311260 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:25.411844 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:25.411815 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:25.512373 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:25.512315 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:25.612987 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:25.612951 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:25.621054 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.621036 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 12:02:25.621199 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.621174 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 12:02:25.656195 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.656176 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 12:02:25.713036 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:25.713005 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:25.714098 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.714074 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 11:57:24 +0000 UTC" deadline="2028-05-15 02:19:03.460602403 +0000 UTC" Sep 09 12:02:25.714146 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.714098 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14726h16m37.746506508s" Sep 09 12:02:25.813937 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:25.813881 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found" Sep 09 12:02:25.858450 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.858428 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 12:02:25.882130 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.882101 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" Sep 09 12:02:25.901597 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.901577 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 12:02:25.902421 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.902409 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" Sep 09 12:02:25.916299 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.916283 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 12:02:25.941509 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.941495 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 12:02:25.970499 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:25.970477 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 12:02:26.472912 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.472885 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 12:02:26.660654 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.660627 2556 apiserver.go:52] "Watching apiserver" Sep 09 12:02:26.667581 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.667552 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 12:02:26.667906 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.667885 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-kgsmn","openshift-multus/network-metrics-daemon-rktvs","openshift-network-operator/iptables-alerter-l6krt","kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8pwxh","openshift-dns/node-resolver-mbf5s","openshift-image-registry/node-ca-6dx5z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal","openshift-multus/multus-additional-cni-plugins-8cpgt","openshift-network-diagnostics/network-check-target-b67ss","openshift-ovn-kubernetes/ovnkube-node-lncg4","kube-system/global-pull-secret-syncer-kzvf5","kube-system/konnectivity-agent-qzjgn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp"] Sep 09 12:02:26.670323 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.670302 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:26.672593 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.672571 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.674972 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.674942 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6dx5z" Sep 09 12:02:26.675523 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.675504 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 12:02:26.675641 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.675625 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xnr4c\"" Sep 09 12:02:26.675787 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.675770 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 12:02:26.675870 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.675855 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r4cnh\"" Sep 09 12:02:26.676076 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.676057 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 12:02:26.676137 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.676077 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 12:02:26.676188 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.676060 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 12:02:26.677149 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.677130 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.679711 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.679696 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbf5s" Sep 09 12:02:26.680358 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.680338 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 12:02:26.680554 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.680540 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 12:02:26.680634 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.680607 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 12:02:26.680697 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.680634 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 12:02:26.681831 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.681572 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.682232 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.682197 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 12:02:26.683981 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.683963 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.686280 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.686261 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:26.686375 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:26.686346 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:26.686438 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.686414 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 12:02:26.686890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.686871 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cb4lp\"" Sep 09 12:02:26.686978 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.686926 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9l6hw\"" Sep 09 12:02:26.687095 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.687078 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 12:02:26.691788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.688814 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.691788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.691753 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-socket-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.691947 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.691786 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vcs\" (UniqueName: \"kubernetes.io/projected/c6c86f8c-76fb-47ba-9189-e02f1bfb574b-kube-api-access-47vcs\") pod \"node-ca-6dx5z\" (UID: \"c6c86f8c-76fb-47ba-9189-e02f1bfb574b\") " pod="openshift-image-registry/node-ca-6dx5z" Sep 09 12:02:26.691947 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.691854 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-lib-modules\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.691947 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.691885 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-var-lib-kubelet\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.692104 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.691931 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvz4\" (UniqueName: \"kubernetes.io/projected/43684e89-2077-4f0a-8aa8-fe6c149d8c16-kube-api-access-dsvz4\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.692104 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692030 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.692104 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692080 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.692258 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692138 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-multus-cni-dir\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.692258 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692168 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad4e6005-53a2-457e-8d11-5fdf78c799b2-agent-certs\") pod \"konnectivity-agent-qzjgn\" (UID: \"ad4e6005-53a2-457e-8d11-5fdf78c799b2\") " pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:26.692258 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692195 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6c86f8c-76fb-47ba-9189-e02f1bfb574b-host\") pod \"node-ca-6dx5z\" (UID: \"c6c86f8c-76fb-47ba-9189-e02f1bfb574b\") " pod="openshift-image-registry/node-ca-6dx5z" Sep 09 12:02:26.692406 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692254 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-modprobe-d\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.692406 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692318 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-run\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.692406 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692352 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-cnibin\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.692406 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692377 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px74f\" (UniqueName: \"kubernetes.io/projected/0e051d05-2b45-4348-86a0-0e9e2a31514f-kube-api-access-px74f\") pod \"node-resolver-mbf5s\" (UID: \"0e051d05-2b45-4348-86a0-0e9e2a31514f\") " pod="openshift-dns/node-resolver-mbf5s" Sep 09 12:02:26.692595 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692411 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/210607a5-ef5e-40d2-a314-03329bcb4064-cni-binary-copy\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.692595 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692453 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-multus-socket-dir-parent\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.692595 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692517 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-host\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.692595 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692554 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-tuned\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.692595 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692592 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.692826 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692621 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6c86f8c-76fb-47ba-9189-e02f1bfb574b-serviceca\") pod \"node-ca-6dx5z\" (UID: \"c6c86f8c-76fb-47ba-9189-e02f1bfb574b\") " pod="openshift-image-registry/node-ca-6dx5z" Sep 09 12:02:26.692826 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692709 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-etc-kubernetes\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.692826 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692761 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshvp\" (UniqueName: \"kubernetes.io/projected/210607a5-ef5e-40d2-a314-03329bcb4064-kube-api-access-qshvp\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.692965 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692798 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-registration-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.692965 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692900 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-kubernetes\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.692965 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692933 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-system-cni-dir\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.693116 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.692961 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e051d05-2b45-4348-86a0-0e9e2a31514f-tmp-dir\") pod \"node-resolver-mbf5s\" (UID: \"0e051d05-2b45-4348-86a0-0e9e2a31514f\") " pod="openshift-dns/node-resolver-mbf5s" Sep 09 12:02:26.693116 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693037 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-cnibin\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.693116 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693065 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-run-k8s-cni-cncf-io\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.693282 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693121 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6rm\" (UniqueName: \"kubernetes.io/projected/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-kube-api-access-9d6rm\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.693282 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693177 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43684e89-2077-4f0a-8aa8-fe6c149d8c16-tmp\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.693282 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693221 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-cni-binary-copy\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.693282 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693250 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4dvp\" (UniqueName: \"kubernetes.io/projected/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-kube-api-access-b4dvp\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.693282 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693276 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-var-lib-cni-bin\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.693517 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-sysctl-d\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.693517 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693495 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-sysctl-conf\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.693615 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693531 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e051d05-2b45-4348-86a0-0e9e2a31514f-hosts-file\") pod \"node-resolver-mbf5s\" (UID: \"0e051d05-2b45-4348-86a0-0e9e2a31514f\") " pod="openshift-dns/node-resolver-mbf5s" Sep 09 12:02:26.693615 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693580 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-var-lib-cni-multus\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.693714 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693604 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-run-multus-certs\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.693714 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693644 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-sys\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.693714 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693697 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.693857 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693726 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-device-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.693857 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693756 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-sys-fs\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.693857 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693783 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-sysconfig\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.693857 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693808 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-systemd\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.693857 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693843 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-var-lib-kubelet\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.694099 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693921 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/210607a5-ef5e-40d2-a314-03329bcb4064-multus-daemon-config\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.694099 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.693963 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad4e6005-53a2-457e-8d11-5fdf78c799b2-konnectivity-ca\") pod \"konnectivity-agent-qzjgn\" (UID: \"ad4e6005-53a2-457e-8d11-5fdf78c799b2\") " pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:26.694099 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.694010 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.694099 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.694049 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-system-cni-dir\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.694099 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.694077 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-run-netns\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.694359 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.694138 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-hostroot\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.694359 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.694170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-multus-conf-dir\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.696005 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.695960 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:26.696087 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:26.696022 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:26.698286 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.697883 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l6krt" Sep 09 12:02:26.700193 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.700174 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:26.700301 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:26.700251 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:26.705697 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.705655 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-flcn8\"" Sep 09 12:02:26.705805 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.705711 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 12:02:26.706440 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.706091 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9w9g5\"" Sep 09 12:02:26.706440 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.706107 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 12:02:26.706440 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.706183 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 12:02:26.706440 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.706255 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 12:02:26.706440 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.706304 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 12:02:26.706440 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.706337 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 12:02:26.706440 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.706352 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 12:02:26.706811 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.706568 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 12:02:26.706811 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.706734 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 12:02:26.706946 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.706905 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 12:02:26.707104 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.707073 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 12:02:26.707801 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.707318 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gjc6r\"" Sep 09 12:02:26.707801 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.707432 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 12:02:26.707801 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.707463 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wqlg9\"" Sep 09 12:02:26.707801 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.707553 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jbd4f\"" Sep 09 12:02:26.707801 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.707596 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 12:02:26.707801 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.707696 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 12:02:26.707801 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.707705 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 12:02:26.707801 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.707716 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 12:02:26.717492 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.717472 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 12:02:26.784655 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.784623 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 12:02:26.795324 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795299 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-run-multus-certs\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.795439 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795336 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-var-lib-kubelet\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.795439 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795362 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/210607a5-ef5e-40d2-a314-03329bcb4064-multus-daemon-config\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.795439 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795385 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad4e6005-53a2-457e-8d11-5fdf78c799b2-konnectivity-ca\") pod \"konnectivity-agent-qzjgn\" (UID: \"ad4e6005-53a2-457e-8d11-5fdf78c799b2\") " pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:26.795439 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795406 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-system-cni-dir\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.795439 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795433 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pw99\" (UniqueName: \"kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99\") pod \"network-check-target-b67ss\" (UID: \"79debca6-ea95-4e3d-8952-5cf5ed2ab970\") " pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:26.795668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vcs\" (UniqueName: \"kubernetes.io/projected/c6c86f8c-76fb-47ba-9189-e02f1bfb574b-kube-api-access-47vcs\") pod \"node-ca-6dx5z\" (UID: \"c6c86f8c-76fb-47ba-9189-e02f1bfb574b\") " pod="openshift-image-registry/node-ca-6dx5z" Sep 09 12:02:26.795668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795483 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.795668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795509 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-multus-cni-dir\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.795668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795530 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-etc-kubernetes\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.795668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795545 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-run-multus-certs\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.795668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795553 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-run-ovn\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.795668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795598 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6c86f8c-76fb-47ba-9189-e02f1bfb574b-host\") pod \"node-ca-6dx5z\" (UID: \"c6c86f8c-76fb-47ba-9189-e02f1bfb574b\") " pod="openshift-image-registry/node-ca-6dx5z" Sep 09 12:02:26.795668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795614 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-var-lib-kubelet\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.795668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795621 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-run\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.795668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795645 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-cnibin\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795671 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/210607a5-ef5e-40d2-a314-03329bcb4064-cni-binary-copy\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795711 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/556aa98a-d697-4e96-beda-0ae8d694b31d-dbus\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795735 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a151e674-cd4a-460a-bc4d-06d31ce26adc-ovn-node-metrics-cert\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795758 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-host\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795782 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795806 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a151e674-cd4a-460a-bc4d-06d31ce26adc-env-overrides\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795835 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-registration-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795861 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-system-cni-dir\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795886 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-run-k8s-cni-cncf-io\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795909 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-var-lib-cni-bin\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795933 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795960 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6rm\" (UniqueName: \"kubernetes.io/projected/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-kube-api-access-9d6rm\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.795985 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4dvp\" (UniqueName: \"kubernetes.io/projected/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-kube-api-access-b4dvp\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796008 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-slash\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796036 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-run-netns\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.796216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796062 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-node-log\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796084 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-cni-netd\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796106 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a151e674-cd4a-460a-bc4d-06d31ce26adc-ovnkube-config\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796129 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-sysctl-conf\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796152 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f7ae721c-3b8c-4830-80b9-47e887405da8-iptables-alerter-script\") pod \"iptables-alerter-l6krt\" (UID: \"f7ae721c-3b8c-4830-80b9-47e887405da8\") " pod="openshift-network-operator/iptables-alerter-l6krt" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796183 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4j7x\" (UniqueName: \"kubernetes.io/projected/f7ae721c-3b8c-4830-80b9-47e887405da8-kube-api-access-q4j7x\") pod \"iptables-alerter-l6krt\" (UID: \"f7ae721c-3b8c-4830-80b9-47e887405da8\") " pod="openshift-network-operator/iptables-alerter-l6krt" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796220 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-sys\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796252 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796259 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796276 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-device-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796307 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-sys-fs\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796348 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-sysconfig\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796372 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-systemd\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796394 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-kubelet\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796420 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796446 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-run-netns\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796469 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-hostroot\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.796980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796495 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-multus-conf-dir\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796518 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-cni-bin\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796542 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-socket-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796566 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-lib-modules\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796595 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-var-lib-kubelet\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796606 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/210607a5-ef5e-40d2-a314-03329bcb4064-multus-daemon-config\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796635 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvz4\" (UniqueName: \"kubernetes.io/projected/43684e89-2077-4f0a-8aa8-fe6c149d8c16-kube-api-access-dsvz4\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796661 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796772 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-multus-cni-dir\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796809 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad4e6005-53a2-457e-8d11-5fdf78c799b2-konnectivity-ca\") pod \"konnectivity-agent-qzjgn\" (UID: \"ad4e6005-53a2-457e-8d11-5fdf78c799b2\") " pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796810 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-multus-socket-dir-parent\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796852 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-multus-socket-dir-parent\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796897 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-run-systemd\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796930 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-etc-kubernetes\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796929 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a151e674-cd4a-460a-bc4d-06d31ce26adc-ovnkube-script-lib\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.796990 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad4e6005-53a2-457e-8d11-5fdf78c799b2-agent-certs\") pod \"konnectivity-agent-qzjgn\" (UID: \"ad4e6005-53a2-457e-8d11-5fdf78c799b2\") " pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797018 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-modprobe-d\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.797832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797046 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px74f\" (UniqueName: \"kubernetes.io/projected/0e051d05-2b45-4348-86a0-0e9e2a31514f-kube-api-access-px74f\") pod \"node-resolver-mbf5s\" (UID: \"0e051d05-2b45-4348-86a0-0e9e2a31514f\") " pod="openshift-dns/node-resolver-mbf5s" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797070 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshvp\" (UniqueName: \"kubernetes.io/projected/210607a5-ef5e-40d2-a314-03329bcb4064-kube-api-access-qshvp\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797066 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-system-cni-dir\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797132 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797139 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7ae721c-3b8c-4830-80b9-47e887405da8-host-slash\") pod \"iptables-alerter-l6krt\" (UID: \"f7ae721c-3b8c-4830-80b9-47e887405da8\") " pod="openshift-network-operator/iptables-alerter-l6krt" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn54v\" (UniqueName: \"kubernetes.io/projected/a151e674-cd4a-460a-bc4d-06d31ce26adc-kube-api-access-zn54v\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797284 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6c86f8c-76fb-47ba-9189-e02f1bfb574b-host\") pod \"node-ca-6dx5z\" (UID: \"c6c86f8c-76fb-47ba-9189-e02f1bfb574b\") " pod="openshift-image-registry/node-ca-6dx5z" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797351 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-run\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797397 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-device-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797420 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-cnibin\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797492 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-sys-fs\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797522 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-hostroot\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797542 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-multus-conf-dir\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797603 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-systemd\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797667 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-socket-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797815 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-lib-modules\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797853 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-var-lib-kubelet\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797880 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.798639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.797954 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/210607a5-ef5e-40d2-a314-03329bcb4064-cni-binary-copy\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798059 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798125 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-run-netns\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798157 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-system-cni-dir\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798192 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-run-k8s-cni-cncf-io\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798235 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-sysconfig\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798309 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-var-lib-cni-bin\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798307 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-host\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798366 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wsvn\" (UniqueName: \"kubernetes.io/projected/40b400cd-b876-4e32-b6c1-a1322eb20817-kube-api-access-4wsvn\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798400 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-registration-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798408 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-tuned\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798437 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798472 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6c86f8c-76fb-47ba-9189-e02f1bfb574b-serviceca\") pod \"node-ca-6dx5z\" (UID: \"c6c86f8c-76fb-47ba-9189-e02f1bfb574b\") " pod="openshift-image-registry/node-ca-6dx5z" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798499 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/556aa98a-d697-4e96-beda-0ae8d694b31d-kubelet-config\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798539 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-var-lib-openvswitch\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798564 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-etc-openvswitch\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798595 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-run-ovn-kubernetes\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.799493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798662 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-kubernetes\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798690 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e051d05-2b45-4348-86a0-0e9e2a31514f-tmp-dir\") pod \"node-resolver-mbf5s\" (UID: \"0e051d05-2b45-4348-86a0-0e9e2a31514f\") " pod="openshift-dns/node-resolver-mbf5s" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798715 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-cnibin\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798901 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-run-openvswitch\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.798989 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43684e89-2077-4f0a-8aa8-fe6c149d8c16-tmp\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799080 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-cni-binary-copy\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799120 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-systemd-units\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799157 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-log-socket\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799183 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-sysctl-d\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799259 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e051d05-2b45-4348-86a0-0e9e2a31514f-hosts-file\") pod \"node-resolver-mbf5s\" (UID: \"0e051d05-2b45-4348-86a0-0e9e2a31514f\") " pod="openshift-dns/node-resolver-mbf5s" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799283 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799302 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-var-lib-cni-multus\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799333 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-sysctl-conf\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799431 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-host-var-lib-cni-multus\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799535 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/210607a5-ef5e-40d2-a314-03329bcb4064-cnibin\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799544 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-sys\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799600 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-kubernetes\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799617 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e051d05-2b45-4348-86a0-0e9e2a31514f-hosts-file\") pod \"node-resolver-mbf5s\" (UID: \"0e051d05-2b45-4348-86a0-0e9e2a31514f\") " pod="openshift-dns/node-resolver-mbf5s" Sep 09 12:02:26.800293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799657 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-sysctl-d\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.801004 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799690 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.801004 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799722 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.801004 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.799901 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-modprobe-d\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.801004 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.800040 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jvzqp" Sep 09 12:02:26.801004 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.800090 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6c86f8c-76fb-47ba-9189-e02f1bfb574b-serviceca\") pod \"node-ca-6dx5z\" (UID: \"c6c86f8c-76fb-47ba-9189-e02f1bfb574b\") " pod="openshift-image-registry/node-ca-6dx5z" Sep 09 12:02:26.801004 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.800184 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e051d05-2b45-4348-86a0-0e9e2a31514f-tmp-dir\") pod \"node-resolver-mbf5s\" (UID: \"0e051d05-2b45-4348-86a0-0e9e2a31514f\") " pod="openshift-dns/node-resolver-mbf5s" Sep 09 12:02:26.801004 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.800340 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-cni-binary-copy\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.803391 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.803369 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43684e89-2077-4f0a-8aa8-fe6c149d8c16-etc-tuned\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.803476 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.803444 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43684e89-2077-4f0a-8aa8-fe6c149d8c16-tmp\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.804263 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.804241 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad4e6005-53a2-457e-8d11-5fdf78c799b2-agent-certs\") pod \"konnectivity-agent-qzjgn\" (UID: \"ad4e6005-53a2-457e-8d11-5fdf78c799b2\") " pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:26.813365 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.813345 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jvzqp" Sep 09 12:02:26.817048 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.816927 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" event={"ID":"968730223ffcdc713c2b8b2489f9ba2f","Type":"ContainerStarted","Data":"657240a2079aebbaa765585b4cc4f8ef38e3dbd64cae5dde7e2a561d4066cf74"} Sep 09 12:02:26.823310 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.823271 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vcs\" (UniqueName: \"kubernetes.io/projected/c6c86f8c-76fb-47ba-9189-e02f1bfb574b-kube-api-access-47vcs\") pod \"node-ca-6dx5z\" (UID: \"c6c86f8c-76fb-47ba-9189-e02f1bfb574b\") " pod="openshift-image-registry/node-ca-6dx5z" Sep 09 12:02:26.823420 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.823274 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvz4\" (UniqueName: \"kubernetes.io/projected/43684e89-2077-4f0a-8aa8-fe6c149d8c16-kube-api-access-dsvz4\") pod \"tuned-8pwxh\" (UID: \"43684e89-2077-4f0a-8aa8-fe6c149d8c16\") " pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:26.829266 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.829242 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4dvp\" (UniqueName: \"kubernetes.io/projected/166bdd7f-1ca5-459d-adfa-a0f76bd75a57-kube-api-access-b4dvp\") pod \"multus-additional-cni-plugins-8cpgt\" (UID: \"166bdd7f-1ca5-459d-adfa-a0f76bd75a57\") " pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:26.830298 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.830269 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px74f\" (UniqueName: \"kubernetes.io/projected/0e051d05-2b45-4348-86a0-0e9e2a31514f-kube-api-access-px74f\") pod \"node-resolver-mbf5s\" (UID: \"0e051d05-2b45-4348-86a0-0e9e2a31514f\") " pod="openshift-dns/node-resolver-mbf5s" Sep 09 12:02:26.842954 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.842932 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6rm\" (UniqueName: \"kubernetes.io/projected/6890a545-9ac7-48a9-98d3-a6ce8feb74c4-kube-api-access-9d6rm\") pod \"aws-ebs-csi-driver-node-5xmhp\" (UID: \"6890a545-9ac7-48a9-98d3-a6ce8feb74c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.847631 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.847613 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshvp\" (UniqueName: \"kubernetes.io/projected/210607a5-ef5e-40d2-a314-03329bcb4064-kube-api-access-qshvp\") pod \"multus-kgsmn\" (UID: \"210607a5-ef5e-40d2-a314-03329bcb4064\") " pod="openshift-multus/multus-kgsmn" Sep 09 12:02:26.899603 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899576 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-kubelet\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.899603 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899605 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-cni-bin\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.899780 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899625 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-run-systemd\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.899780 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899648 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a151e674-cd4a-460a-bc4d-06d31ce26adc-ovnkube-script-lib\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.899780 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899674 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7ae721c-3b8c-4830-80b9-47e887405da8-host-slash\") pod \"iptables-alerter-l6krt\" (UID: \"f7ae721c-3b8c-4830-80b9-47e887405da8\") " pod="openshift-network-operator/iptables-alerter-l6krt" Sep 09 12:02:26.899780 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899697 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn54v\" (UniqueName: \"kubernetes.io/projected/a151e674-cd4a-460a-bc4d-06d31ce26adc-kube-api-access-zn54v\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.899780 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899706 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-run-systemd\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.899780 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899720 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:26.899780 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899743 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wsvn\" (UniqueName: \"kubernetes.io/projected/40b400cd-b876-4e32-b6c1-a1322eb20817-kube-api-access-4wsvn\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:26.899780 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899749 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-cni-bin\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.899780 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899673 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-kubelet\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.899780 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899770 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/556aa98a-d697-4e96-beda-0ae8d694b31d-kubelet-config\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899793 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-var-lib-openvswitch\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899814 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-etc-openvswitch\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899837 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-run-ovn-kubernetes\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899862 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-run-openvswitch\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899884 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-systemd-units\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-log-socket\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899936 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pw99\" (UniqueName: \"kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99\") pod \"network-check-target-b67ss\" (UID: \"79debca6-ea95-4e3d-8952-5cf5ed2ab970\") " pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899963 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-run-ovn\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.899988 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/556aa98a-d697-4e96-beda-0ae8d694b31d-dbus\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900011 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a151e674-cd4a-460a-bc4d-06d31ce26adc-ovn-node-metrics-cert\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900037 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900061 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a151e674-cd4a-460a-bc4d-06d31ce26adc-env-overrides\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900089 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900116 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-slash\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900138 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-run-netns\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900162 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-node-log\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900191 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-cni-netd\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900233 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a151e674-cd4a-460a-bc4d-06d31ce26adc-ovnkube-config\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900259 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f7ae721c-3b8c-4830-80b9-47e887405da8-iptables-alerter-script\") pod \"iptables-alerter-l6krt\" (UID: \"f7ae721c-3b8c-4830-80b9-47e887405da8\") " pod="openshift-network-operator/iptables-alerter-l6krt" Sep 09 12:02:26.900680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900283 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4j7x\" (UniqueName: \"kubernetes.io/projected/f7ae721c-3b8c-4830-80b9-47e887405da8-kube-api-access-q4j7x\") pod \"iptables-alerter-l6krt\" (UID: \"f7ae721c-3b8c-4830-80b9-47e887405da8\") " pod="openshift-network-operator/iptables-alerter-l6krt" Sep 09 12:02:26.900680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900329 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a151e674-cd4a-460a-bc4d-06d31ce26adc-ovnkube-script-lib\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900378 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7ae721c-3b8c-4830-80b9-47e887405da8-host-slash\") pod \"iptables-alerter-l6krt\" (UID: \"f7ae721c-3b8c-4830-80b9-47e887405da8\") " pod="openshift-network-operator/iptables-alerter-l6krt" Sep 09 12:02:26.900680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900418 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-run-ovn\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.900680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900576 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/556aa98a-d697-4e96-beda-0ae8d694b31d-dbus\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:26.900680 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:26.900638 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:26.901120 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:26.900700 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs podName:40b400cd-b876-4e32-b6c1-a1322eb20817 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:27.400677266 +0000 UTC m=+3.080583450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs") pod "network-metrics-daemon-rktvs" (UID: "40b400cd-b876-4e32-b6c1-a1322eb20817") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:26.901120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900894 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-run-openvswitch\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900913 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-systemd-units\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900954 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-var-lib-openvswitch\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900968 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-run-netns\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900995 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-etc-openvswitch\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.900994 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/556aa98a-d697-4e96-beda-0ae8d694b31d-kubelet-config\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:26.901120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.901040 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-run-ovn-kubernetes\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.901049 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-log-socket\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.901100 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-slash\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901652 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:26.901129 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:26.901652 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:26.901176 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret podName:556aa98a-d697-4e96-beda-0ae8d694b31d nodeName:}" failed. No retries permitted until 2026-09-09 12:02:27.401160038 +0000 UTC m=+3.081066207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret") pod "global-pull-secret-syncer-kzvf5" (UID: "556aa98a-d697-4e96-beda-0ae8d694b31d") : object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:26.901652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.901233 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.901264 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-host-cni-netd\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.901283 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a151e674-cd4a-460a-bc4d-06d31ce26adc-node-log\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.901443 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a151e674-cd4a-460a-bc4d-06d31ce26adc-env-overrides\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901982 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.901718 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a151e674-cd4a-460a-bc4d-06d31ce26adc-ovnkube-config\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.901982 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.901799 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f7ae721c-3b8c-4830-80b9-47e887405da8-iptables-alerter-script\") pod \"iptables-alerter-l6krt\" (UID: \"f7ae721c-3b8c-4830-80b9-47e887405da8\") " pod="openshift-network-operator/iptables-alerter-l6krt" Sep 09 12:02:26.902893 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.902871 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a151e674-cd4a-460a-bc4d-06d31ce26adc-ovn-node-metrics-cert\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.937345 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.937316 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn54v\" (UniqueName: \"kubernetes.io/projected/a151e674-cd4a-460a-bc4d-06d31ce26adc-kube-api-access-zn54v\") pod \"ovnkube-node-lncg4\" (UID: \"a151e674-cd4a-460a-bc4d-06d31ce26adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:26.938396 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:26.938375 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 12:02:26.938396 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:26.938399 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 12:02:26.938617 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:26.938413 2556 projected.go:196] Error preparing data for projected volume kube-api-access-8pw99 for pod openshift-network-diagnostics/network-check-target-b67ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:26.938617 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.938434 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wsvn\" (UniqueName: \"kubernetes.io/projected/40b400cd-b876-4e32-b6c1-a1322eb20817-kube-api-access-4wsvn\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:26.938617 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:26.938475 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99 podName:79debca6-ea95-4e3d-8952-5cf5ed2ab970 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:27.438458038 +0000 UTC m=+3.118364207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8pw99" (UniqueName: "kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99") pod "network-check-target-b67ss" (UID: "79debca6-ea95-4e3d-8952-5cf5ed2ab970") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:26.939295 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.939262 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4j7x\" (UniqueName: \"kubernetes.io/projected/f7ae721c-3b8c-4830-80b9-47e887405da8-kube-api-access-q4j7x\") pod \"iptables-alerter-l6krt\" (UID: \"f7ae721c-3b8c-4830-80b9-47e887405da8\") " pod="openshift-network-operator/iptables-alerter-l6krt" Sep 09 12:02:26.983682 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.983630 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:26.991462 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:26.991441 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" Sep 09 12:02:26.994223 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:26.994187 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4e6005_53a2_457e_8d11_5fdf78c799b2.slice/crio-2193174f4955d0dbce9346f044b197a87d87f792558324de3a5c0c1f6d2b05e9 WatchSource:0}: Error finding container 2193174f4955d0dbce9346f044b197a87d87f792558324de3a5c0c1f6d2b05e9: Status 404 returned error can't find the container with id 2193174f4955d0dbce9346f044b197a87d87f792558324de3a5c0c1f6d2b05e9 Sep 09 12:02:27.000237 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:27.000196 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6890a545_9ac7_48a9_98d3_a6ce8feb74c4.slice/crio-c50fe4bf8cd319685642ff78a0e762008b0cfa5b4a3a19627385dd601250d614 WatchSource:0}: Error finding container c50fe4bf8cd319685642ff78a0e762008b0cfa5b4a3a19627385dd601250d614: Status 404 returned error can't find the container with id c50fe4bf8cd319685642ff78a0e762008b0cfa5b4a3a19627385dd601250d614 Sep 09 12:02:27.003568 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.003550 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6dx5z" Sep 09 12:02:27.009141 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.009123 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" Sep 09 12:02:27.013073 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:27.013041 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c86f8c_76fb_47ba_9189_e02f1bfb574b.slice/crio-6355aedab3a9fe592c9e4dd27970c0ac0ca0faff23d9a0f694bc56d38fef7506 WatchSource:0}: Error finding container 6355aedab3a9fe592c9e4dd27970c0ac0ca0faff23d9a0f694bc56d38fef7506: Status 404 returned error can't find the container with id 6355aedab3a9fe592c9e4dd27970c0ac0ca0faff23d9a0f694bc56d38fef7506 Sep 09 12:02:27.016183 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.016162 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbf5s" Sep 09 12:02:27.020244 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:27.020190 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43684e89_2077_4f0a_8aa8_fe6c149d8c16.slice/crio-c3fc9cbc1648acc85599fa9aa46134b7ecf50c9380ceda954a00f618ffaa69d6 WatchSource:0}: Error finding container c3fc9cbc1648acc85599fa9aa46134b7ecf50c9380ceda954a00f618ffaa69d6: Status 404 returned error can't find the container with id c3fc9cbc1648acc85599fa9aa46134b7ecf50c9380ceda954a00f618ffaa69d6 Sep 09 12:02:27.023920 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.023901 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kgsmn" Sep 09 12:02:27.024738 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:27.024719 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e051d05_2b45_4348_86a0_0e9e2a31514f.slice/crio-832ded9e034314e9f92b20e29c0894e8f250705ca0f2507aa759bcddc0f7f997 WatchSource:0}: Error finding container 832ded9e034314e9f92b20e29c0894e8f250705ca0f2507aa759bcddc0f7f997: Status 404 returned error can't find the container with id 832ded9e034314e9f92b20e29c0894e8f250705ca0f2507aa759bcddc0f7f997 Sep 09 12:02:27.031344 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.031323 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8cpgt" Sep 09 12:02:27.039458 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.039438 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:27.046905 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.046887 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l6krt" Sep 09 12:02:27.403741 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.403648 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:27.403741 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.403710 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:27.403939 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:27.403820 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:27.403939 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:27.403826 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:27.403939 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:27.403877 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs podName:40b400cd-b876-4e32-b6c1-a1322eb20817 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:28.403859521 +0000 UTC m=+4.083765686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs") pod "network-metrics-daemon-rktvs" (UID: "40b400cd-b876-4e32-b6c1-a1322eb20817") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:27.403939 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:27.403897 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret podName:556aa98a-d697-4e96-beda-0ae8d694b31d nodeName:}" failed. No retries permitted until 2026-09-09 12:02:28.403886949 +0000 UTC m=+4.083793114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret") pod "global-pull-secret-syncer-kzvf5" (UID: "556aa98a-d697-4e96-beda-0ae8d694b31d") : object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:27.503980 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.503941 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pw99\" (UniqueName: \"kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99\") pod \"network-check-target-b67ss\" (UID: \"79debca6-ea95-4e3d-8952-5cf5ed2ab970\") " pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:27.504154 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:27.504089 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 12:02:27.504154 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:27.504112 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 12:02:27.504154 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:27.504120 2556 projected.go:196] Error preparing data for projected volume kube-api-access-8pw99 for pod openshift-network-diagnostics/network-check-target-b67ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:27.504291 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:27.504176 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99 podName:79debca6-ea95-4e3d-8952-5cf5ed2ab970 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:28.504160556 +0000 UTC m=+4.184066721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pw99" (UniqueName: "kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99") pod "network-check-target-b67ss" (UID: "79debca6-ea95-4e3d-8952-5cf5ed2ab970") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:27.661519 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:27.661304 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210607a5_ef5e_40d2_a314_03329bcb4064.slice/crio-f4dbb32047a157b28b730d7a9da742d9c5ba770095b394a7085759a15222a114 WatchSource:0}: Error finding container f4dbb32047a157b28b730d7a9da742d9c5ba770095b394a7085759a15222a114: Status 404 returned error can't find the container with id f4dbb32047a157b28b730d7a9da742d9c5ba770095b394a7085759a15222a114 Sep 09 12:02:27.662767 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:27.662743 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166bdd7f_1ca5_459d_adfa_a0f76bd75a57.slice/crio-5e91b67b52cdc0c98f03ef468e2a83a85846207b8915e54920e1b375a97a7780 WatchSource:0}: Error finding container 5e91b67b52cdc0c98f03ef468e2a83a85846207b8915e54920e1b375a97a7780: Status 404 returned error can't find the container with id 5e91b67b52cdc0c98f03ef468e2a83a85846207b8915e54920e1b375a97a7780 Sep 09 12:02:27.663694 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:27.663597 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ae721c_3b8c_4830_80b9_47e887405da8.slice/crio-8eff6359e059d9e66f8f6854313ddef0490c06b562d9873a0b75f2785c3f4aad WatchSource:0}: Error finding container 8eff6359e059d9e66f8f6854313ddef0490c06b562d9873a0b75f2785c3f4aad: Status 404 returned error can't find the container with id 8eff6359e059d9e66f8f6854313ddef0490c06b562d9873a0b75f2785c3f4aad Sep 09 12:02:27.664278 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:27.664256 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda151e674_cd4a_460a_bc4d_06d31ce26adc.slice/crio-c387158acb4c1e04873dc7d9d596f0363bd57859f9675d27c9be0e2e158abc39 WatchSource:0}: Error finding container c387158acb4c1e04873dc7d9d596f0363bd57859f9675d27c9be0e2e158abc39: Status 404 returned error can't find the container with id c387158acb4c1e04873dc7d9d596f0363bd57859f9675d27c9be0e2e158abc39 Sep 09 12:02:27.811456 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.811422 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:27.811595 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:27.811557 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:27.817536 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.817469 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 11:57:26 +0000 UTC" deadline="2028-03-13 15:57:31.934914359 +0000 UTC" Sep 09 12:02:27.817536 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.817513 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13227h55m4.117405023s" Sep 09 12:02:27.821760 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.821737 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" event={"ID":"968730223ffcdc713c2b8b2489f9ba2f","Type":"ContainerStarted","Data":"f016f95886894f104b9c557f01c8636033563c70353459d4797e8e21b53e5499"} Sep 09 12:02:27.823527 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.823504 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l6krt" event={"ID":"f7ae721c-3b8c-4830-80b9-47e887405da8","Type":"ContainerStarted","Data":"8eff6359e059d9e66f8f6854313ddef0490c06b562d9873a0b75f2785c3f4aad"} Sep 09 12:02:27.825044 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.825021 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cpgt" event={"ID":"166bdd7f-1ca5-459d-adfa-a0f76bd75a57","Type":"ContainerStarted","Data":"5e91b67b52cdc0c98f03ef468e2a83a85846207b8915e54920e1b375a97a7780"} Sep 09 12:02:27.826935 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.826917 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" event={"ID":"43684e89-2077-4f0a-8aa8-fe6c149d8c16","Type":"ContainerStarted","Data":"c3fc9cbc1648acc85599fa9aa46134b7ecf50c9380ceda954a00f618ffaa69d6"} Sep 09 12:02:27.828302 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.828260 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6dx5z" event={"ID":"c6c86f8c-76fb-47ba-9189-e02f1bfb574b","Type":"ContainerStarted","Data":"6355aedab3a9fe592c9e4dd27970c0ac0ca0faff23d9a0f694bc56d38fef7506"} Sep 09 12:02:27.829632 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.829605 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" event={"ID":"6890a545-9ac7-48a9-98d3-a6ce8feb74c4","Type":"ContainerStarted","Data":"c50fe4bf8cd319685642ff78a0e762008b0cfa5b4a3a19627385dd601250d614"} Sep 09 12:02:27.831279 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.831255 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzjgn" event={"ID":"ad4e6005-53a2-457e-8d11-5fdf78c799b2","Type":"ContainerStarted","Data":"2193174f4955d0dbce9346f044b197a87d87f792558324de3a5c0c1f6d2b05e9"} Sep 09 12:02:27.832674 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.832648 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" event={"ID":"a151e674-cd4a-460a-bc4d-06d31ce26adc","Type":"ContainerStarted","Data":"c387158acb4c1e04873dc7d9d596f0363bd57859f9675d27c9be0e2e158abc39"} Sep 09 12:02:27.834409 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.834363 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kgsmn" event={"ID":"210607a5-ef5e-40d2-a314-03329bcb4064","Type":"ContainerStarted","Data":"f4dbb32047a157b28b730d7a9da742d9c5ba770095b394a7085759a15222a114"} Sep 09 12:02:27.835799 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.835776 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbf5s" event={"ID":"0e051d05-2b45-4348-86a0-0e9e2a31514f","Type":"ContainerStarted","Data":"832ded9e034314e9f92b20e29c0894e8f250705ca0f2507aa759bcddc0f7f997"} Sep 09 12:02:27.841390 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:27.841345 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" podStartSLOduration=2.841327743 podStartE2EDuration="2.841327743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:02:27.840394959 +0000 UTC m=+3.520301150" watchObservedRunningTime="2026-09-09 12:02:27.841327743 +0000 UTC m=+3.521233932" Sep 09 12:02:28.297809 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:02:28.297775 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca52691c2cdddfad83976f50a01762b.slice/crio-04d5121c65a630930c984ab1939c8d47db9a78899bf24363190d9aee6e616af7 WatchSource:0}: Error finding container 04d5121c65a630930c984ab1939c8d47db9a78899bf24363190d9aee6e616af7: Status 404 returned error can't find the container with id 04d5121c65a630930c984ab1939c8d47db9a78899bf24363190d9aee6e616af7 Sep 09 12:02:28.410645 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:28.410612 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:28.410818 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:28.410774 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:28.410884 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:28.410842 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret podName:556aa98a-d697-4e96-beda-0ae8d694b31d nodeName:}" failed. No retries permitted until 2026-09-09 12:02:30.410823476 +0000 UTC m=+6.090729654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret") pod "global-pull-secret-syncer-kzvf5" (UID: "556aa98a-d697-4e96-beda-0ae8d694b31d") : object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:28.411183 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:28.411161 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:28.411347 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:28.411333 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:28.411403 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:28.411384 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs podName:40b400cd-b876-4e32-b6c1-a1322eb20817 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:30.411370209 +0000 UTC m=+6.091276378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs") pod "network-metrics-daemon-rktvs" (UID: "40b400cd-b876-4e32-b6c1-a1322eb20817") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:28.511682 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:28.511572 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pw99\" (UniqueName: \"kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99\") pod \"network-check-target-b67ss\" (UID: \"79debca6-ea95-4e3d-8952-5cf5ed2ab970\") " pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:28.511848 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:28.511738 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 12:02:28.511848 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:28.511758 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 12:02:28.511848 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:28.511770 2556 projected.go:196] Error preparing data for projected volume kube-api-access-8pw99 for pod openshift-network-diagnostics/network-check-target-b67ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:28.511848 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:28.511823 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99 podName:79debca6-ea95-4e3d-8952-5cf5ed2ab970 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:30.511805245 +0000 UTC m=+6.191711417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pw99" (UniqueName: "kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99") pod "network-check-target-b67ss" (UID: "79debca6-ea95-4e3d-8952-5cf5ed2ab970") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:28.814824 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:28.814039 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:28.814824 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:28.814162 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:28.814824 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:28.814584 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:28.814824 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:28.814687 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:28.818690 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:28.818618 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 11:57:26 +0000 UTC" deadline="2028-05-20 20:51:57.021424238 +0000 UTC" Sep 09 12:02:28.818690 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:28.818647 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14864h49m28.202781155s" Sep 09 12:02:28.847247 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:28.847222 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" event={"ID":"1ca52691c2cdddfad83976f50a01762b","Type":"ContainerStarted","Data":"04d5121c65a630930c984ab1939c8d47db9a78899bf24363190d9aee6e616af7"} Sep 09 12:02:29.811086 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:29.811056 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:29.811266 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:29.811178 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:30.427167 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:30.427129 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:30.427769 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:30.427223 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:30.427769 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:30.427369 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:30.427769 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:30.427428 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs podName:40b400cd-b876-4e32-b6c1-a1322eb20817 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:34.427411907 +0000 UTC m=+10.107318084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs") pod "network-metrics-daemon-rktvs" (UID: "40b400cd-b876-4e32-b6c1-a1322eb20817") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:30.427943 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:30.427898 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:30.427995 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:30.427947 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret podName:556aa98a-d697-4e96-beda-0ae8d694b31d nodeName:}" failed. No retries permitted until 2026-09-09 12:02:34.427932815 +0000 UTC m=+10.107838986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret") pod "global-pull-secret-syncer-kzvf5" (UID: "556aa98a-d697-4e96-beda-0ae8d694b31d") : object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:30.528220 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:30.527594 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pw99\" (UniqueName: \"kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99\") pod \"network-check-target-b67ss\" (UID: \"79debca6-ea95-4e3d-8952-5cf5ed2ab970\") " pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:30.528220 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:30.527785 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 12:02:30.528220 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:30.527806 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 12:02:30.528220 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:30.527820 2556 projected.go:196] Error preparing data for projected volume kube-api-access-8pw99 for pod openshift-network-diagnostics/network-check-target-b67ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:30.528220 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:30.527876 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99 podName:79debca6-ea95-4e3d-8952-5cf5ed2ab970 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:34.527858799 +0000 UTC m=+10.207764969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pw99" (UniqueName: "kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99") pod "network-check-target-b67ss" (UID: "79debca6-ea95-4e3d-8952-5cf5ed2ab970") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:30.810868 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:30.810786 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:30.811017 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:30.810936 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:30.811351 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:30.811327 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:30.811463 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:30.811436 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:31.813421 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:31.813373 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:31.813878 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:31.813511 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:32.812339 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:32.812308 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:32.812524 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:32.812436 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:32.812904 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:32.812866 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:32.812998 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:32.812969 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:33.811665 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:33.811632 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:33.812117 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:33.811761 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:34.461647 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:34.461608 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:34.461799 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:34.461690 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:34.461873 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:34.461844 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:34.461922 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:34.461908 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret podName:556aa98a-d697-4e96-beda-0ae8d694b31d nodeName:}" failed. No retries permitted until 2026-09-09 12:02:42.461889211 +0000 UTC m=+18.141795377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret") pod "global-pull-secret-syncer-kzvf5" (UID: "556aa98a-d697-4e96-beda-0ae8d694b31d") : object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:34.462413 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:34.462325 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:34.462413 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:34.462380 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs podName:40b400cd-b876-4e32-b6c1-a1322eb20817 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:42.462364345 +0000 UTC m=+18.142270515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs") pod "network-metrics-daemon-rktvs" (UID: "40b400cd-b876-4e32-b6c1-a1322eb20817") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:34.562823 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:34.562789 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pw99\" (UniqueName: \"kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99\") pod \"network-check-target-b67ss\" (UID: \"79debca6-ea95-4e3d-8952-5cf5ed2ab970\") " pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:34.562999 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:34.562960 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 12:02:34.562999 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:34.562988 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 12:02:34.563112 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:34.563001 2556 projected.go:196] Error preparing data for projected volume kube-api-access-8pw99 for pod openshift-network-diagnostics/network-check-target-b67ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:34.563112 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:34.563062 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99 podName:79debca6-ea95-4e3d-8952-5cf5ed2ab970 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:42.563043229 +0000 UTC m=+18.242949414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pw99" (UniqueName: "kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99") pod "network-check-target-b67ss" (UID: "79debca6-ea95-4e3d-8952-5cf5ed2ab970") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:34.813277 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:34.812784 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:34.813277 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:34.812886 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:34.814119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:34.813952 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:34.814119 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:34.814077 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:35.811338 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:35.811305 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:35.811519 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:35.811429 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:36.811234 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:36.811146 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:36.811652 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:36.811316 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:36.811652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:36.811371 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:36.811652 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:36.811485 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:37.811198 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:37.811167 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:37.811351 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:37.811289 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:38.810902 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:38.810874 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:38.810902 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:38.810904 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:38.811134 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:38.810986 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:38.811134 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:38.811113 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:39.810736 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:39.810700 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:39.811170 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:39.810817 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:40.810863 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:40.810782 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:40.811305 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:40.810910 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:40.811305 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:40.810977 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:40.811305 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:40.811100 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:41.810932 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:41.810889 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:41.811528 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:41.811015 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:42.519709 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:42.519675 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:42.519896 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:42.519745 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:42.519896 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:42.519837 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:42.519896 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:42.519856 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:42.520057 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:42.519914 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret podName:556aa98a-d697-4e96-beda-0ae8d694b31d nodeName:}" failed. No retries permitted until 2026-09-09 12:02:58.519893117 +0000 UTC m=+34.199799302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret") pod "global-pull-secret-syncer-kzvf5" (UID: "556aa98a-d697-4e96-beda-0ae8d694b31d") : object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:42.520057 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:42.519933 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs podName:40b400cd-b876-4e32-b6c1-a1322eb20817 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:58.519925334 +0000 UTC m=+34.199831499 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs") pod "network-metrics-daemon-rktvs" (UID: "40b400cd-b876-4e32-b6c1-a1322eb20817") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:42.620560 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:42.620516 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pw99\" (UniqueName: \"kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99\") pod \"network-check-target-b67ss\" (UID: \"79debca6-ea95-4e3d-8952-5cf5ed2ab970\") " pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:42.620732 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:42.620693 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 12:02:42.620732 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:42.620719 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 12:02:42.620732 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:42.620732 2556 projected.go:196] Error preparing data for projected volume kube-api-access-8pw99 for pod openshift-network-diagnostics/network-check-target-b67ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:42.620889 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:42.620797 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99 podName:79debca6-ea95-4e3d-8952-5cf5ed2ab970 nodeName:}" failed. No retries permitted until 2026-09-09 12:02:58.620777766 +0000 UTC m=+34.300683944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pw99" (UniqueName: "kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99") pod "network-check-target-b67ss" (UID: "79debca6-ea95-4e3d-8952-5cf5ed2ab970") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:42.810770 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:42.810687 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:42.810931 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:42.810687 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:42.810931 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:42.810818 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:42.810931 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:42.810910 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:43.811043 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:43.811008 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:43.811410 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:43.811111 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:44.812046 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:44.812014 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:44.812459 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:44.812125 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:44.812459 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:44.812219 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:44.812459 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:44.812320 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:45.811401 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.811245 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:45.811501 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:45.811484 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:45.883005 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.882980 2556 generic.go:356] "Generic (PLEG): container finished" podID="166bdd7f-1ca5-459d-adfa-a0f76bd75a57" containerID="c97420962494a960368d2a25cc66f52af3fbb3ab1dc99fc8f5a7439c2c8605a4" exitCode=0 Sep 09 12:02:45.883601 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.883062 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cpgt" event={"ID":"166bdd7f-1ca5-459d-adfa-a0f76bd75a57","Type":"ContainerDied","Data":"c97420962494a960368d2a25cc66f52af3fbb3ab1dc99fc8f5a7439c2c8605a4"} Sep 09 12:02:45.884659 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.884642 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" event={"ID":"43684e89-2077-4f0a-8aa8-fe6c149d8c16","Type":"ContainerStarted","Data":"28998bd2ccfafe781c76dae40b239b697baaec45a1990d24ba2530ff176988db"} Sep 09 12:02:45.886253 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.886229 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6dx5z" event={"ID":"c6c86f8c-76fb-47ba-9189-e02f1bfb574b","Type":"ContainerStarted","Data":"cce93e86cb79c0134c100dea95f4e805ad2f4f64442b8ba2400b0433477dacf7"} Sep 09 12:02:45.887668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.887648 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" event={"ID":"6890a545-9ac7-48a9-98d3-a6ce8feb74c4","Type":"ContainerStarted","Data":"216b94bc95e52632207b841439822df5f53b5a9891a7c1d9f69ce2698e5bc08c"} Sep 09 12:02:45.888925 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.888896 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzjgn" event={"ID":"ad4e6005-53a2-457e-8d11-5fdf78c799b2","Type":"ContainerStarted","Data":"44c202a5c08a2f9ade8b6c80e4287813eeb6ce4e0ad721a02b8d63cd589032b0"} Sep 09 12:02:45.891327 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.891308 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" event={"ID":"a151e674-cd4a-460a-bc4d-06d31ce26adc","Type":"ContainerStarted","Data":"4760c467329db20bf709a55e3f481e2796ae1b138d3ac53076a46970a3d4434c"} Sep 09 12:02:45.891407 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.891376 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" event={"ID":"a151e674-cd4a-460a-bc4d-06d31ce26adc","Type":"ContainerStarted","Data":"ea35a8f54852f27d4b5a59ca673c154238f77929c18fc26ec1a986043995bef6"} Sep 09 12:02:45.891456 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.891408 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" event={"ID":"a151e674-cd4a-460a-bc4d-06d31ce26adc","Type":"ContainerStarted","Data":"3e9acdc52e28e8e8aec66f427fd70692fe7fa4412f51f551b7effcce393e04cd"} Sep 09 12:02:45.891456 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.891436 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" event={"ID":"a151e674-cd4a-460a-bc4d-06d31ce26adc","Type":"ContainerStarted","Data":"bef07d8f83f7a45e9c44c4fd384c5a101067f5a158a124a257efc91e8a0be516"} Sep 09 12:02:45.891538 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.891462 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" event={"ID":"a151e674-cd4a-460a-bc4d-06d31ce26adc","Type":"ContainerStarted","Data":"6e22f1c43d84403c4a6cffaf25faa6da5f6a101b42e2870098645a918947dbd5"} Sep 09 12:02:45.892537 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.892519 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kgsmn" event={"ID":"210607a5-ef5e-40d2-a314-03329bcb4064","Type":"ContainerStarted","Data":"684c2a0b13a4835fdf543ad86a236fe9a36ba853adbd4356176394229fd9c78f"} Sep 09 12:02:45.893791 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.893775 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbf5s" event={"ID":"0e051d05-2b45-4348-86a0-0e9e2a31514f","Type":"ContainerStarted","Data":"2a5bfe8b19acde8b44df05779b9312914cf44958df7f8fcc4aba8b0e1e225acf"} Sep 09 12:02:45.895109 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.895091 2556 generic.go:356] "Generic (PLEG): container finished" podID="1ca52691c2cdddfad83976f50a01762b" containerID="a2fda1db6073777f3d0d5e6682fc4e384e8712a946b6759200f1546f0454b6f2" exitCode=0 Sep 09 12:02:45.895176 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:45.895117 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" event={"ID":"1ca52691c2cdddfad83976f50a01762b","Type":"ContainerDied","Data":"a2fda1db6073777f3d0d5e6682fc4e384e8712a946b6759200f1546f0454b6f2"} Sep 09 12:02:46.173350 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.173297 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qzjgn" podStartSLOduration=5.4372458869999996 podStartE2EDuration="22.173280578s" totalImagesPullingTime="16.736034691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:02:46.09665756 +0000 UTC m=+21.776563747" watchObservedRunningTime="2026-09-09 12:02:46.173280578 +0000 UTC m=+21.853186759" Sep 09 12:02:46.173687 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.173659 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8pwxh" podStartSLOduration=2.973604678 podStartE2EDuration="21.173650064s" totalImagesPullingTime="18.200045386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:02:46.173433486 +0000 UTC m=+21.853339689" watchObservedRunningTime="2026-09-09 12:02:46.173650064 +0000 UTC m=+21.853556251" Sep 09 12:02:46.244144 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.244094 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:46.244712 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.244687 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:46.256711 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.256664 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kgsmn" podStartSLOduration=3.494625584 podStartE2EDuration="21.256647096s" totalImagesPullingTime="17.762021512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:02:46.231400584 +0000 UTC m=+21.911306786" watchObservedRunningTime="2026-09-09 12:02:46.256647096 +0000 UTC m=+21.936553284" Sep 09 12:02:46.257105 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.257075 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6dx5z" podStartSLOduration=3.097735836 podStartE2EDuration="21.257066526s" totalImagesPullingTime="18.15933069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:02:46.256625062 +0000 UTC m=+21.936531248" watchObservedRunningTime="2026-09-09 12:02:46.257066526 +0000 UTC m=+21.936972694" Sep 09 12:02:46.287573 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.287533 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mbf5s" podStartSLOduration=3.137370261 podStartE2EDuration="21.287514928s" totalImagesPullingTime="18.150144667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:02:46.287449301 +0000 UTC m=+21.967355489" watchObservedRunningTime="2026-09-09 12:02:46.287514928 +0000 UTC m=+21.967421116" Sep 09 12:02:46.810664 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.810640 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:46.810764 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.810688 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:46.810804 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:46.810785 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:46.810935 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:46.810915 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:46.832355 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.832330 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 12:02:46.899278 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.899235 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l6krt" event={"ID":"f7ae721c-3b8c-4830-80b9-47e887405da8","Type":"ContainerStarted","Data":"13c46a2de805c7675c7b07715b29d481d87ce13f7d84b38fe38f51b88af33d01"} Sep 09 12:02:46.901862 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.901829 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" event={"ID":"6890a545-9ac7-48a9-98d3-a6ce8feb74c4","Type":"ContainerStarted","Data":"dcbb0531cfb3475c8b3a40d0b5fad2ec28cca2cbfebbd72ae43bcca6d605d906"} Sep 09 12:02:46.905575 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.905490 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" event={"ID":"a151e674-cd4a-460a-bc4d-06d31ce26adc","Type":"ContainerStarted","Data":"88b9453edd23a87d67010f14f726870f4fbed62d9f53ce6310339fa7b4f6a76d"} Sep 09 12:02:46.906262 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.906238 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:46.906391 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.906344 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qzjgn" Sep 09 12:02:46.920933 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:46.920885 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l6krt" podStartSLOduration=4.410152833 podStartE2EDuration="21.920867013s" totalImagesPullingTime="17.51071418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:02:46.920713465 +0000 UTC m=+22.600619655" watchObservedRunningTime="2026-09-09 12:02:46.920867013 +0000 UTC m=+22.600773203" Sep 09 12:02:47.777441 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:47.777097 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T12:02:46.832348762Z","UUID":"1b0d198a-2d8a-48ec-bab6-869ad584196f","Handler":null,"Name":"","Endpoint":""} Sep 09 12:02:47.779256 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:47.779234 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 12:02:47.779391 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:47.779270 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 12:02:47.811187 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:47.811160 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:47.811355 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:47.811309 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:47.909158 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:47.909123 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" event={"ID":"1ca52691c2cdddfad83976f50a01762b","Type":"ContainerStarted","Data":"94a1681add0e7079fbd0f445bdc2b6f81b2b56de0167a4b57d78801790c7c1e9"} Sep 09 12:02:47.945413 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:47.945362 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" podStartSLOduration=22.945348 podStartE2EDuration="22.945348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:02:47.944280877 +0000 UTC m=+23.624187063" watchObservedRunningTime="2026-09-09 12:02:47.945348 +0000 UTC m=+23.625254187" Sep 09 12:02:48.810900 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:48.810864 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:48.810900 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:48.810902 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:48.811160 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:48.811022 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:48.811160 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:48.811134 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:48.913709 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:48.913675 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" event={"ID":"6890a545-9ac7-48a9-98d3-a6ce8feb74c4","Type":"ContainerStarted","Data":"9ac1e1ba2d93b39cc5c375f0a7d64a5935db1ca62ee664f1bb1c8cb0cbb3aaca"} Sep 09 12:02:48.917427 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:48.917400 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" event={"ID":"a151e674-cd4a-460a-bc4d-06d31ce26adc","Type":"ContainerStarted","Data":"a38c22d31cb13d9f99e5755e4d335166551fb58ad449b478fca8c106ed7e778d"} Sep 09 12:02:48.939751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:48.939704 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xmhp" podStartSLOduration=4.433922996 podStartE2EDuration="24.939684421s" totalImagesPullingTime="20.505761425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:02:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:02:48.939564264 +0000 UTC m=+24.619470452" watchObservedRunningTime="2026-09-09 12:02:48.939684421 +0000 UTC m=+24.619590613" Sep 09 12:02:49.810644 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:49.810613 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:49.810818 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:49.810746 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:50.811059 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:50.810877 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:50.811416 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:50.810877 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:50.811416 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:50.811130 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:50.811416 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:50.811187 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:50.925672 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:50.925640 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" event={"ID":"a151e674-cd4a-460a-bc4d-06d31ce26adc","Type":"ContainerStarted","Data":"3132195af46529ff435f16b2a62196cc88c73c5a8401d2100c7f41ea74ed71d6"} Sep 09 12:02:50.925902 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:50.925880 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:50.927494 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:50.927468 2556 generic.go:356] "Generic (PLEG): container finished" podID="166bdd7f-1ca5-459d-adfa-a0f76bd75a57" containerID="2809d14ef8732ae8f13502e0767a21c7ea9e322e32f4ce9086d8bee7cdf6fc86" exitCode=0 Sep 09 12:02:50.927610 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:50.927501 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cpgt" event={"ID":"166bdd7f-1ca5-459d-adfa-a0f76bd75a57","Type":"ContainerDied","Data":"2809d14ef8732ae8f13502e0767a21c7ea9e322e32f4ce9086d8bee7cdf6fc86"} Sep 09 12:02:50.941212 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:50.941181 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:50.972981 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:50.972932 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" podStartSLOduration=8.398927662 podStartE2EDuration="25.972919935s" totalImagesPullingTime="17.573992273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:02:50.972755293 +0000 UTC m=+26.652661474" watchObservedRunningTime="2026-09-09 12:02:50.972919935 +0000 UTC m=+26.652826125" Sep 09 12:02:51.811146 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:51.811107 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:51.811584 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:51.811234 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:51.930138 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:51.930111 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:51.930259 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:51.930194 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:51.947426 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:51.947405 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:02:52.811101 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:52.811070 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:52.811347 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:52.811233 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:52.811347 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:52.811259 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:52.811347 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:52.811339 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:52.933772 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:52.933744 2556 generic.go:356] "Generic (PLEG): container finished" podID="166bdd7f-1ca5-459d-adfa-a0f76bd75a57" containerID="b2c47584872a5c5836e4f4be54fab9847bd8a81457e9e853748531bc75ad027d" exitCode=0 Sep 09 12:02:52.933941 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:52.933817 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cpgt" event={"ID":"166bdd7f-1ca5-459d-adfa-a0f76bd75a57","Type":"ContainerDied","Data":"b2c47584872a5c5836e4f4be54fab9847bd8a81457e9e853748531bc75ad027d"} Sep 09 12:02:53.811452 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:53.811430 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:53.811873 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:53.811520 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:53.938603 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:53.938573 2556 generic.go:356] "Generic (PLEG): container finished" podID="166bdd7f-1ca5-459d-adfa-a0f76bd75a57" containerID="ea261c23888d487bebb1d834eea01638ad9c03ec293863b87971632ccd83df08" exitCode=0 Sep 09 12:02:53.938769 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:53.938654 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cpgt" event={"ID":"166bdd7f-1ca5-459d-adfa-a0f76bd75a57","Type":"ContainerDied","Data":"ea261c23888d487bebb1d834eea01638ad9c03ec293863b87971632ccd83df08"} Sep 09 12:02:54.812430 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:54.812398 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:54.812785 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:54.812507 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:54.812785 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:54.812551 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:54.812785 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:54.812636 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:55.810878 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:55.810850 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:55.811020 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:55.810969 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:56.811172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:56.811137 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:56.811624 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:56.811284 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:56.811702 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:56.811649 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:56.811775 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:56.811749 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:57.811489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:57.811460 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:57.811890 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:57.811585 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:02:58.534558 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:58.534522 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:58.534841 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:58.534586 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:58.534841 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:58.534692 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:58.534841 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:58.534692 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:58.534841 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:58.534756 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs podName:40b400cd-b876-4e32-b6c1-a1322eb20817 nodeName:}" failed. No retries permitted until 2026-09-09 12:03:30.53473624 +0000 UTC m=+66.214642408 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs") pod "network-metrics-daemon-rktvs" (UID: "40b400cd-b876-4e32-b6c1-a1322eb20817") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 12:02:58.534841 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:58.534775 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret podName:556aa98a-d697-4e96-beda-0ae8d694b31d nodeName:}" failed. No retries permitted until 2026-09-09 12:03:30.534765188 +0000 UTC m=+66.214671358 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret") pod "global-pull-secret-syncer-kzvf5" (UID: "556aa98a-d697-4e96-beda-0ae8d694b31d") : object "kube-system"/"original-pull-secret" not registered Sep 09 12:02:58.635001 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:58.634970 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pw99\" (UniqueName: \"kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99\") pod \"network-check-target-b67ss\" (UID: \"79debca6-ea95-4e3d-8952-5cf5ed2ab970\") " pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:58.635144 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:58.635121 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 12:02:58.635144 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:58.635139 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 12:02:58.635238 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:58.635148 2556 projected.go:196] Error preparing data for projected volume kube-api-access-8pw99 for pod openshift-network-diagnostics/network-check-target-b67ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:58.635238 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:58.635198 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99 podName:79debca6-ea95-4e3d-8952-5cf5ed2ab970 nodeName:}" failed. No retries permitted until 2026-09-09 12:03:30.635181677 +0000 UTC m=+66.315087847 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pw99" (UniqueName: "kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99") pod "network-check-target-b67ss" (UID: "79debca6-ea95-4e3d-8952-5cf5ed2ab970") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 12:02:58.811343 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:58.811260 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:02:58.811343 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:58.811287 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:02:58.811553 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:58.811396 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:02:58.811553 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:58.811524 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:02:59.810807 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:02:59.810775 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:02:59.810986 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:02:59.810894 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:03:00.810948 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:00.810879 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:00.811309 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:00.810988 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:03:00.811309 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:00.811044 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:00.811309 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:00.811139 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:03:00.956067 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:00.956035 2556 generic.go:356] "Generic (PLEG): container finished" podID="166bdd7f-1ca5-459d-adfa-a0f76bd75a57" containerID="473c404fd730fc3f9a4f7baccc460705b249411fbbb8bd6a084efaa56d597698" exitCode=0 Sep 09 12:03:00.956261 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:00.956091 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cpgt" event={"ID":"166bdd7f-1ca5-459d-adfa-a0f76bd75a57","Type":"ContainerDied","Data":"473c404fd730fc3f9a4f7baccc460705b249411fbbb8bd6a084efaa56d597698"} Sep 09 12:03:01.811288 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:01.811250 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:01.811672 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:01.811354 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:03:01.961694 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:01.961662 2556 generic.go:356] "Generic (PLEG): container finished" podID="166bdd7f-1ca5-459d-adfa-a0f76bd75a57" containerID="eae057274c3953583be9d4e89c546d51c3024651644c7a41bbab46d2639e60bd" exitCode=0 Sep 09 12:03:01.961849 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:01.961722 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cpgt" event={"ID":"166bdd7f-1ca5-459d-adfa-a0f76bd75a57","Type":"ContainerDied","Data":"eae057274c3953583be9d4e89c546d51c3024651644c7a41bbab46d2639e60bd"} Sep 09 12:03:02.811577 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:02.811327 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:02.812004 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:02.811374 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:02.812004 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:02.811688 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:03:02.812004 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:02.811770 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:03:02.967604 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:02.967573 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cpgt" event={"ID":"166bdd7f-1ca5-459d-adfa-a0f76bd75a57","Type":"ContainerStarted","Data":"6b01c7f88eaf3f7943246b0b94b98916bf519264735c4f168fc1b561e3669761"} Sep 09 12:03:03.810729 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:03.810693 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:03.810891 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:03.810837 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:03:04.811954 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:04.811917 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:04.812394 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:04.811968 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:04.812394 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:04.812048 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:03:04.812394 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:04.812179 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:03:05.810562 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:05.810533 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:05.810742 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:05.810625 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:03:06.811214 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:06.811163 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:06.811214 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:06.811195 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:06.811691 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:06.811323 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:03:06.811691 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:06.811326 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:03:07.810847 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:07.810815 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:07.811015 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:07.810913 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:03:08.811520 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:08.811484 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:08.811884 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:08.811614 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:03:08.811884 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:08.811658 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:08.811884 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:08.811747 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:03:09.811428 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:09.811398 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:09.811648 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:09.811565 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:03:10.811096 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:10.811047 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:10.811271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:10.811101 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:10.811271 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:10.811189 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:03:10.811397 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:10.811323 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:03:10.864754 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:10.864615 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8cpgt" podStartSLOduration=15.401315496 podStartE2EDuration="45.864567299s" totalImagesPullingTime="30.463251803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:03.005020514 +0000 UTC m=+38.684926700" watchObservedRunningTime="2026-09-09 12:03:10.864567299 +0000 UTC m=+46.544473484" Sep 09 12:03:10.865650 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:10.865628 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kzvf5"] Sep 09 12:03:10.865765 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:10.865738 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:10.865874 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:10.865849 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:03:10.876908 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:10.876876 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b67ss"] Sep 09 12:03:10.880319 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:10.880298 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rktvs"] Sep 09 12:03:10.984519 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:10.984492 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:10.984673 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:10.984492 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:10.984673 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:10.984580 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:03:10.984745 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:10.984670 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:03:12.810609 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:12.810583 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:12.810609 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:12.810596 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:12.811119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:12.810605 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:12.811119 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:12.810700 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:03:12.811119 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:12.810779 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:03:12.811119 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:12.810861 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:03:14.812778 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:14.812747 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:14.813400 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:14.812861 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:14.813509 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:14.813489 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzvf5" podUID="556aa98a-d697-4e96-beda-0ae8d694b31d" Sep 09 12:03:14.813583 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:14.813377 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rktvs" podUID="40b400cd-b876-4e32-b6c1-a1322eb20817" Sep 09 12:03:14.813583 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:14.812927 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:14.813677 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:14.813610 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b67ss" podUID="79debca6-ea95-4e3d-8952-5cf5ed2ab970" Sep 09 12:03:15.045273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.045031 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeReady" Sep 09 12:03:15.045412 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.045323 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 12:03:15.123815 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.123787 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-96bsw"] Sep 09 12:03:15.134658 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.134635 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r564d"] Sep 09 12:03:15.134794 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.134764 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.138151 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.138127 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 12:03:15.138391 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.138344 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t4scp\"" Sep 09 12:03:15.138477 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.138390 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 12:03:15.138539 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.138484 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 12:03:15.139099 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.139086 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 12:03:15.147681 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.147558 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.148488 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.148471 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-96bsw"] Sep 09 12:03:15.156670 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.156646 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rt2j7\"" Sep 09 12:03:15.156904 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.156885 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 12:03:15.157233 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.157197 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 12:03:15.157612 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.157590 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r564d"] Sep 09 12:03:15.216739 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.216709 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9blz8"] Sep 09 12:03:15.228183 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.228166 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9blz8" Sep 09 12:03:15.231829 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.231801 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 12:03:15.232123 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.232105 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 12:03:15.232226 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.232123 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 12:03:15.232316 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.232288 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zt888\"" Sep 09 12:03:15.232762 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.232742 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9blz8"] Sep 09 12:03:15.254104 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.254074 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f65bb66e-9ad6-4ebb-9261-3688257fecaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.254104 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.254107 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584516fc-2718-462e-8753-491fa88197d0-tmp-dir\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.254295 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.254171 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f65bb66e-9ad6-4ebb-9261-3688257fecaf-crio-socket\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.254295 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.254226 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584516fc-2718-462e-8753-491fa88197d0-metrics-tls\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.254295 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.254283 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f65bb66e-9ad6-4ebb-9261-3688257fecaf-data-volume\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.254388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.254328 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vv4\" (UniqueName: \"kubernetes.io/projected/f65bb66e-9ad6-4ebb-9261-3688257fecaf-kube-api-access-47vv4\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.254388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.254353 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f65bb66e-9ad6-4ebb-9261-3688257fecaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.254388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.254371 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/584516fc-2718-462e-8753-491fa88197d0-config-volume\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.254476 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.254386 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6fb\" (UniqueName: \"kubernetes.io/projected/584516fc-2718-462e-8753-491fa88197d0-kube-api-access-2w6fb\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.355745 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.355669 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f65bb66e-9ad6-4ebb-9261-3688257fecaf-crio-socket\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.355745 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.355704 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584516fc-2718-462e-8753-491fa88197d0-metrics-tls\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.355745 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.355725 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4cl\" (UniqueName: \"kubernetes.io/projected/3a3d2c78-1cae-4271-8944-0437fd466a3b-kube-api-access-cj4cl\") pod \"ingress-canary-9blz8\" (UID: \"3a3d2c78-1cae-4271-8944-0437fd466a3b\") " pod="openshift-ingress-canary/ingress-canary-9blz8" Sep 09 12:03:15.355918 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.355755 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a3d2c78-1cae-4271-8944-0437fd466a3b-cert\") pod \"ingress-canary-9blz8\" (UID: \"3a3d2c78-1cae-4271-8944-0437fd466a3b\") " pod="openshift-ingress-canary/ingress-canary-9blz8" Sep 09 12:03:15.355918 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.355852 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f65bb66e-9ad6-4ebb-9261-3688257fecaf-data-volume\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.355918 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.355899 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vv4\" (UniqueName: \"kubernetes.io/projected/f65bb66e-9ad6-4ebb-9261-3688257fecaf-kube-api-access-47vv4\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.356026 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.355917 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f65bb66e-9ad6-4ebb-9261-3688257fecaf-crio-socket\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.356026 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.355929 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f65bb66e-9ad6-4ebb-9261-3688257fecaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.356026 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.355954 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/584516fc-2718-462e-8753-491fa88197d0-config-volume\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.356026 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.355982 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6fb\" (UniqueName: \"kubernetes.io/projected/584516fc-2718-462e-8753-491fa88197d0-kube-api-access-2w6fb\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.356238 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.356027 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f65bb66e-9ad6-4ebb-9261-3688257fecaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.356238 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.356048 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584516fc-2718-462e-8753-491fa88197d0-tmp-dir\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.356238 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.356118 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f65bb66e-9ad6-4ebb-9261-3688257fecaf-data-volume\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.356440 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.356421 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584516fc-2718-462e-8753-491fa88197d0-tmp-dir\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.356608 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.356589 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/584516fc-2718-462e-8753-491fa88197d0-config-volume\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.356689 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.356613 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f65bb66e-9ad6-4ebb-9261-3688257fecaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.359588 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.359567 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584516fc-2718-462e-8753-491fa88197d0-metrics-tls\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.359692 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.359571 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f65bb66e-9ad6-4ebb-9261-3688257fecaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.366281 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.366258 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vv4\" (UniqueName: \"kubernetes.io/projected/f65bb66e-9ad6-4ebb-9261-3688257fecaf-kube-api-access-47vv4\") pod \"insights-runtime-extractor-96bsw\" (UID: \"f65bb66e-9ad6-4ebb-9261-3688257fecaf\") " pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.366395 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.366292 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6fb\" (UniqueName: \"kubernetes.io/projected/584516fc-2718-462e-8753-491fa88197d0-kube-api-access-2w6fb\") pod \"dns-default-r564d\" (UID: \"584516fc-2718-462e-8753-491fa88197d0\") " pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.444331 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.444292 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-96bsw" Sep 09 12:03:15.456476 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.456442 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r564d" Sep 09 12:03:15.456594 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.456555 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4cl\" (UniqueName: \"kubernetes.io/projected/3a3d2c78-1cae-4271-8944-0437fd466a3b-kube-api-access-cj4cl\") pod \"ingress-canary-9blz8\" (UID: \"3a3d2c78-1cae-4271-8944-0437fd466a3b\") " pod="openshift-ingress-canary/ingress-canary-9blz8" Sep 09 12:03:15.456594 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.456584 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a3d2c78-1cae-4271-8944-0437fd466a3b-cert\") pod \"ingress-canary-9blz8\" (UID: \"3a3d2c78-1cae-4271-8944-0437fd466a3b\") " pod="openshift-ingress-canary/ingress-canary-9blz8" Sep 09 12:03:15.459001 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.458981 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a3d2c78-1cae-4271-8944-0437fd466a3b-cert\") pod \"ingress-canary-9blz8\" (UID: \"3a3d2c78-1cae-4271-8944-0437fd466a3b\") " pod="openshift-ingress-canary/ingress-canary-9blz8" Sep 09 12:03:15.466125 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.466105 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4cl\" (UniqueName: \"kubernetes.io/projected/3a3d2c78-1cae-4271-8944-0437fd466a3b-kube-api-access-cj4cl\") pod \"ingress-canary-9blz8\" (UID: \"3a3d2c78-1cae-4271-8944-0437fd466a3b\") " pod="openshift-ingress-canary/ingress-canary-9blz8" Sep 09 12:03:15.537047 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.536637 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9blz8" Sep 09 12:03:15.756322 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.756294 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r564d"] Sep 09 12:03:15.760133 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:15.760108 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584516fc_2718_462e_8753_491fa88197d0.slice/crio-7544eb81b6e16b85909aed60672dd0d8160d56552ba18a5b20036b9be1d8c166 WatchSource:0}: Error finding container 7544eb81b6e16b85909aed60672dd0d8160d56552ba18a5b20036b9be1d8c166: Status 404 returned error can't find the container with id 7544eb81b6e16b85909aed60672dd0d8160d56552ba18a5b20036b9be1d8c166 Sep 09 12:03:15.763944 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.763914 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-96bsw"] Sep 09 12:03:15.767630 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:15.767607 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65bb66e_9ad6_4ebb_9261_3688257fecaf.slice/crio-3614c63a956755aa6fbd761cd40b9285d80dc5e9cf04c2bcd86861270d5618f7 WatchSource:0}: Error finding container 3614c63a956755aa6fbd761cd40b9285d80dc5e9cf04c2bcd86861270d5618f7: Status 404 returned error can't find the container with id 3614c63a956755aa6fbd761cd40b9285d80dc5e9cf04c2bcd86861270d5618f7 Sep 09 12:03:15.842050 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.842025 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9blz8"] Sep 09 12:03:15.845849 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:15.845828 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3d2c78_1cae_4271_8944_0437fd466a3b.slice/crio-7dcabdf6edea5c4e8f45e7ba520a4a8fa16dc56a374380801a095de65c0fb3ee WatchSource:0}: Error finding container 7dcabdf6edea5c4e8f45e7ba520a4a8fa16dc56a374380801a095de65c0fb3ee: Status 404 returned error can't find the container with id 7dcabdf6edea5c4e8f45e7ba520a4a8fa16dc56a374380801a095de65c0fb3ee Sep 09 12:03:15.995551 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.995517 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r564d" event={"ID":"584516fc-2718-462e-8753-491fa88197d0","Type":"ContainerStarted","Data":"7544eb81b6e16b85909aed60672dd0d8160d56552ba18a5b20036b9be1d8c166"} Sep 09 12:03:15.997026 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.997003 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96bsw" event={"ID":"f65bb66e-9ad6-4ebb-9261-3688257fecaf","Type":"ContainerStarted","Data":"d40f508442037f86c318686ada0f3ea20bccc6f961469b48965639ce3719f0d4"} Sep 09 12:03:15.997246 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.997047 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96bsw" event={"ID":"f65bb66e-9ad6-4ebb-9261-3688257fecaf","Type":"ContainerStarted","Data":"3614c63a956755aa6fbd761cd40b9285d80dc5e9cf04c2bcd86861270d5618f7"} Sep 09 12:03:15.998116 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:15.998093 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9blz8" event={"ID":"3a3d2c78-1cae-4271-8944-0437fd466a3b","Type":"ContainerStarted","Data":"7dcabdf6edea5c4e8f45e7ba520a4a8fa16dc56a374380801a095de65c0fb3ee"} Sep 09 12:03:16.686055 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.686020 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dbdb787db-gw7r5"] Sep 09 12:03:16.688427 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.688401 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.706359 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.706335 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 12:03:16.707726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.707017 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 12:03:16.707726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.707071 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 12:03:16.707726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.707170 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 12:03:16.707726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.707169 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 12:03:16.707726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.707341 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mp6gb\"" Sep 09 12:03:16.707726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.707567 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 12:03:16.707726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.707591 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 12:03:16.718584 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.718564 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 12:03:16.724493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.724464 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dbdb787db-gw7r5"] Sep 09 12:03:16.811475 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.811416 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:16.811475 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.811433 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:16.811999 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.811922 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:16.819538 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.819460 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 12:03:16.819709 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.819568 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 12:03:16.819709 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.819705 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 12:03:16.819828 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.819599 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 12:03:16.819828 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.819618 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-85k95\"" Sep 09 12:03:16.819923 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.819655 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8phgx\"" Sep 09 12:03:16.866759 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.866729 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-serving-cert\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.867176 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.866767 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-console-config\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.867176 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.866786 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d7vb\" (UniqueName: \"kubernetes.io/projected/12287520-b3c1-448b-aed9-5b46fb31a568-kube-api-access-9d7vb\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.867176 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.866823 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-oauth-serving-cert\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.867176 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.866859 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-oauth-config\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.867176 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.866901 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-trusted-ca-bundle\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.867176 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.866934 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-service-ca\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.968327 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.968293 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-serving-cert\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.968327 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.968331 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-console-config\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.968548 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.968350 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d7vb\" (UniqueName: \"kubernetes.io/projected/12287520-b3c1-448b-aed9-5b46fb31a568-kube-api-access-9d7vb\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.968548 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.968384 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-oauth-serving-cert\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.968548 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.968474 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-oauth-config\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.968696 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.968539 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-trusted-ca-bundle\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.968696 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.968600 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-service-ca\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.969338 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.969280 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-console-config\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.969489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.969351 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-oauth-serving-cert\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.969489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.969427 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-trusted-ca-bundle\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.969937 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.969916 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-service-ca\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.972103 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.972080 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-oauth-config\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.972263 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.972246 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-serving-cert\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.978945 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.978926 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d7vb\" (UniqueName: \"kubernetes.io/projected/12287520-b3c1-448b-aed9-5b46fb31a568-kube-api-access-9d7vb\") pod \"console-6dbdb787db-gw7r5\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:16.998762 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:16.998734 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:17.003011 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:17.002983 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96bsw" event={"ID":"f65bb66e-9ad6-4ebb-9261-3688257fecaf","Type":"ContainerStarted","Data":"968afb15b40e8889e376efe754f81f8e0556d1f0fe7a9955a108b0f0094d5b5a"} Sep 09 12:03:17.284040 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:17.284007 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dbdb787db-gw7r5"] Sep 09 12:03:17.549759 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:17.549688 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12287520_b3c1_448b_aed9_5b46fb31a568.slice/crio-977402f7fe0633e5a3d930238f7dedbf45607083024b3b6203a440e63162b74a WatchSource:0}: Error finding container 977402f7fe0633e5a3d930238f7dedbf45607083024b3b6203a440e63162b74a: Status 404 returned error can't find the container with id 977402f7fe0633e5a3d930238f7dedbf45607083024b3b6203a440e63162b74a Sep 09 12:03:18.008474 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.008244 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dbdb787db-gw7r5" event={"ID":"12287520-b3c1-448b-aed9-5b46fb31a568","Type":"ContainerStarted","Data":"977402f7fe0633e5a3d930238f7dedbf45607083024b3b6203a440e63162b74a"} Sep 09 12:03:18.010082 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.010056 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r564d" event={"ID":"584516fc-2718-462e-8753-491fa88197d0","Type":"ContainerStarted","Data":"9124ef389a98313a72f65821e1dcaa312bbe2134866a91f24696e6afc602fcca"} Sep 09 12:03:18.114520 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.114494 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg"] Sep 09 12:03:18.116973 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.116950 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.122982 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.122952 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 12:03:18.123217 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.123184 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 09 12:03:18.123467 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.123447 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 12:03:18.123921 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.123663 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vbvt4\"" Sep 09 12:03:18.123921 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.123774 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 09 12:03:18.124250 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.124016 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 12:03:18.129222 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.128750 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg"] Sep 09 12:03:18.131199 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.131182 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.139922 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.139666 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 09 12:03:18.139922 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.139897 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 09 12:03:18.140505 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.140109 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 09 12:03:18.140505 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.140353 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-67dkj\"" Sep 09 12:03:18.158056 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.158029 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg"] Sep 09 12:03:18.158225 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.158103 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg"] Sep 09 12:03:18.178741 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.178699 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.178935 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.178758 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.178935 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.178789 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clv9z\" (UniqueName: \"kubernetes.io/projected/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-api-access-clv9z\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.178935 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.178871 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.178935 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.178907 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e41430b3-e44e-4e0c-8b90-ac215547e15f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.179143 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.178977 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d0a2e6f-20b8-44d8-98cf-9433a0604560-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.179143 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.179011 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d0a2e6f-20b8-44d8-98cf-9433a0604560-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.179143 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.179035 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgjtb\" (UniqueName: \"kubernetes.io/projected/5d0a2e6f-20b8-44d8-98cf-9433a0604560-kube-api-access-fgjtb\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.179143 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.179077 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41430b3-e44e-4e0c-8b90-ac215547e15f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.179143 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.179099 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d0a2e6f-20b8-44d8-98cf-9433a0604560-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.180287 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.180264 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9rxfq"] Sep 09 12:03:18.182491 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.182470 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.187389 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.186833 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 12:03:18.187389 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.187052 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f4f54\"" Sep 09 12:03:18.187389 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.187282 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 12:03:18.187389 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.187348 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.279891 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.279978 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52be1a8c-fc1d-40ab-8294-c75bae298228-sys\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280016 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d0a2e6f-20b8-44d8-98cf-9433a0604560-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280041 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52be1a8c-fc1d-40ab-8294-c75bae298228-root\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280065 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52be1a8c-fc1d-40ab-8294-c75bae298228-metrics-client-ca\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280092 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d0a2e6f-20b8-44d8-98cf-9433a0604560-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280115 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-wtmp\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280141 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280172 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280195 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clv9z\" (UniqueName: \"kubernetes.io/projected/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-api-access-clv9z\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280243 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-tls\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280271 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e41430b3-e44e-4e0c-8b90-ac215547e15f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280304 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d0a2e6f-20b8-44d8-98cf-9433a0604560-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280337 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgjtb\" (UniqueName: \"kubernetes.io/projected/5d0a2e6f-20b8-44d8-98cf-9433a0604560-kube-api-access-fgjtb\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.282502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280376 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n84b\" (UniqueName: \"kubernetes.io/projected/52be1a8c-fc1d-40ab-8294-c75bae298228-kube-api-access-9n84b\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.283273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280416 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41430b3-e44e-4e0c-8b90-ac215547e15f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.283273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280441 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.283273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280477 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.283273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280501 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-textfile\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.283273 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:18.280672 2556 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 09 12:03:18.283273 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:18.280728 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-tls podName:e41430b3-e44e-4e0c-8b90-ac215547e15f nodeName:}" failed. No retries permitted until 2026-09-09 12:03:18.780711282 +0000 UTC m=+54.460617448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-t7nkg" (UID: "e41430b3-e44e-4e0c-8b90-ac215547e15f") : secret "kube-state-metrics-tls" not found Sep 09 12:03:18.283273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.280986 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d0a2e6f-20b8-44d8-98cf-9433a0604560-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.283273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.282322 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41430b3-e44e-4e0c-8b90-ac215547e15f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.283273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.282711 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e41430b3-e44e-4e0c-8b90-ac215547e15f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.283652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.283323 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.286156 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.286111 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.286291 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.286154 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d0a2e6f-20b8-44d8-98cf-9433a0604560-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.286777 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.286448 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d0a2e6f-20b8-44d8-98cf-9433a0604560-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.297313 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.297136 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clv9z\" (UniqueName: \"kubernetes.io/projected/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-api-access-clv9z\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.300091 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.300066 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgjtb\" (UniqueName: \"kubernetes.io/projected/5d0a2e6f-20b8-44d8-98cf-9433a0604560-kube-api-access-fgjtb\") pod \"openshift-state-metrics-657fd6c54-gbngg\" (UID: \"5d0a2e6f-20b8-44d8-98cf-9433a0604560\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.381158 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381128 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n84b\" (UniqueName: \"kubernetes.io/projected/52be1a8c-fc1d-40ab-8294-c75bae298228-kube-api-access-9n84b\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381329 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381192 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-textfile\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381329 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381244 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381329 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381289 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52be1a8c-fc1d-40ab-8294-c75bae298228-sys\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381329 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381316 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52be1a8c-fc1d-40ab-8294-c75bae298228-root\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381532 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381338 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52be1a8c-fc1d-40ab-8294-c75bae298228-metrics-client-ca\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381532 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381363 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-wtmp\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381532 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381389 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381532 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381433 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-tls\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381532 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381510 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52be1a8c-fc1d-40ab-8294-c75bae298228-sys\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381546 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52be1a8c-fc1d-40ab-8294-c75bae298228-root\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381751 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:18.381563 2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 12:03:18.381751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381612 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-textfile\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.381751 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:18.381619 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-tls podName:52be1a8c-fc1d-40ab-8294-c75bae298228 nodeName:}" failed. No retries permitted until 2026-09-09 12:03:18.88160103 +0000 UTC m=+54.561507202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-tls") pod "node-exporter-9rxfq" (UID: "52be1a8c-fc1d-40ab-8294-c75bae298228") : secret "node-exporter-tls" not found Sep 09 12:03:18.381751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.381650 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-wtmp\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.382123 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.382095 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.382265 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.382137 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52be1a8c-fc1d-40ab-8294-c75bae298228-metrics-client-ca\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.384010 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.383966 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.397956 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.397909 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n84b\" (UniqueName: \"kubernetes.io/projected/52be1a8c-fc1d-40ab-8294-c75bae298228-kube-api-access-9n84b\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.431766 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.431737 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" Sep 09 12:03:18.722056 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.721913 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg"] Sep 09 12:03:18.726034 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:18.726007 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0a2e6f_20b8_44d8_98cf_9433a0604560.slice/crio-193e459f65ffac6746a91f90de356cbf25459d7489e88df0f78cc5ebebec5149 WatchSource:0}: Error finding container 193e459f65ffac6746a91f90de356cbf25459d7489e88df0f78cc5ebebec5149: Status 404 returned error can't find the container with id 193e459f65ffac6746a91f90de356cbf25459d7489e88df0f78cc5ebebec5149 Sep 09 12:03:18.785622 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.785583 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.787929 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.787906 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e41430b3-e44e-4e0c-8b90-ac215547e15f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-t7nkg\" (UID: \"e41430b3-e44e-4e0c-8b90-ac215547e15f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:18.887018 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.886988 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-tls\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:18.890130 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:18.890101 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52be1a8c-fc1d-40ab-8294-c75bae298228-node-exporter-tls\") pod \"node-exporter-9rxfq\" (UID: \"52be1a8c-fc1d-40ab-8294-c75bae298228\") " pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:19.016771 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.016684 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r564d" event={"ID":"584516fc-2718-462e-8753-491fa88197d0","Type":"ContainerStarted","Data":"e8aa026cc3d27a474b6cb4ec17f53da265b07bfa6a49ad0669ba7bea0d1f838d"} Sep 09 12:03:19.017407 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.016795 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r564d" Sep 09 12:03:19.019334 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.019307 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96bsw" event={"ID":"f65bb66e-9ad6-4ebb-9261-3688257fecaf","Type":"ContainerStarted","Data":"fb480b48c8985d82a75c0f7554af45089f285d6f3cf25696e8b913ce27fc5dbb"} Sep 09 12:03:19.022105 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.022076 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" event={"ID":"5d0a2e6f-20b8-44d8-98cf-9433a0604560","Type":"ContainerStarted","Data":"bcc3a7338bf93c49badd7e33941e5fcc46911a166a52aae869fa151a49c5d528"} Sep 09 12:03:19.022226 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.022132 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" event={"ID":"5d0a2e6f-20b8-44d8-98cf-9433a0604560","Type":"ContainerStarted","Data":"d2725de2537188f4439a4f6a0ce8a2517611223255a907a038d5e887e523b881"} Sep 09 12:03:19.022226 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.022160 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" event={"ID":"5d0a2e6f-20b8-44d8-98cf-9433a0604560","Type":"ContainerStarted","Data":"193e459f65ffac6746a91f90de356cbf25459d7489e88df0f78cc5ebebec5149"} Sep 09 12:03:19.043649 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.043617 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" Sep 09 12:03:19.046138 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.046097 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r564d" podStartSLOduration=2.231041081 podStartE2EDuration="4.046075309s" totalImagesPullingTime="1.815034228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:19.045150852 +0000 UTC m=+54.725057040" watchObservedRunningTime="2026-09-09 12:03:19.046075309 +0000 UTC m=+54.725981500" Sep 09 12:03:19.095940 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.095700 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9rxfq" Sep 09 12:03:19.183147 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.183091 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-96bsw" podStartSLOduration=2.048574603 podStartE2EDuration="4.183064179s" totalImagesPullingTime="2.134489576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:03:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:19.075030968 +0000 UTC m=+54.754937158" watchObservedRunningTime="2026-09-09 12:03:19.183064179 +0000 UTC m=+54.862970367" Sep 09 12:03:19.183508 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.183446 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 12:03:19.186724 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.186701 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.190740 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.190714 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 12:03:19.191060 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.191041 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 12:03:19.192172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.191309 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 12:03:19.192172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.191605 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 12:03:19.192172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.191823 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 12:03:19.192172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.192032 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 12:03:19.192172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.192036 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 12:03:19.192484 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.192268 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t8bvx\"" Sep 09 12:03:19.192484 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.192443 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 12:03:19.192706 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.192674 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 12:03:19.213866 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.213772 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 12:03:19.290409 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290330 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290409 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290366 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290610 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290422 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290610 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290472 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290610 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290509 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgwpr\" (UniqueName: \"kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-kube-api-access-bgwpr\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290610 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290547 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290610 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290576 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290860 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290602 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-out\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290860 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290639 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290860 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290664 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290860 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290696 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290860 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290731 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.290860 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.290764 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-web-config\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.338286 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.338161 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg"] Sep 09 12:03:19.392122 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.391816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.392122 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.391861 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-out\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.392122 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.391892 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.392122 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.391915 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.392122 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.391946 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.392122 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.391973 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.392122 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.392003 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-web-config\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.392122 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:19.392113 2556 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 09 12:03:19.392623 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:19.392184 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-main-tls podName:b6cb9d4f-b509-49a6-a74b-837b6d5cb721 nodeName:}" failed. No retries permitted until 2026-09-09 12:03:19.892162029 +0000 UTC m=+55.572068214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721") : secret "alertmanager-main-tls" not found Sep 09 12:03:19.393712 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.393687 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.394110 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.394083 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.394238 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.394188 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.399143 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.396512 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-web-config\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.399274 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.399152 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.399934 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.399908 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-out\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.400037 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.400014 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.400328 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.400313 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.400503 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.400490 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.400593 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.400584 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.400665 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.400657 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgwpr\" (UniqueName: \"kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-kube-api-access-bgwpr\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.400738 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.400728 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.403449 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.403395 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.405005 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.404694 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.405005 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.404697 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.405147 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.405069 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.407031 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.406988 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.420035 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.420010 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgwpr\" (UniqueName: \"kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-kube-api-access-bgwpr\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.905994 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.905947 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:19.908687 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:19.908662 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:20.014510 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:20.014467 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41430b3_e44e_4e0c_8b90_ac215547e15f.slice/crio-2ccfafdc41816f91a609a8682271fdeeb0164cd040eca6a84e9c439f2f55a7d4 WatchSource:0}: Error finding container 2ccfafdc41816f91a609a8682271fdeeb0164cd040eca6a84e9c439f2f55a7d4: Status 404 returned error can't find the container with id 2ccfafdc41816f91a609a8682271fdeeb0164cd040eca6a84e9c439f2f55a7d4 Sep 09 12:03:20.026740 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:20.026713 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rxfq" event={"ID":"52be1a8c-fc1d-40ab-8294-c75bae298228","Type":"ContainerStarted","Data":"c582044d0ecfd5bf0dcd7633ae93fb02fd21a2669d8ec9e20e04ec1878bdfd0b"} Sep 09 12:03:20.027966 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:20.027944 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" event={"ID":"e41430b3-e44e-4e0c-8b90-ac215547e15f","Type":"ContainerStarted","Data":"2ccfafdc41816f91a609a8682271fdeeb0164cd040eca6a84e9c439f2f55a7d4"} Sep 09 12:03:20.100493 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:20.100455 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:03:21.262197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.262163 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-65bb57b978-szsdl"] Sep 09 12:03:21.276714 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.276694 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.279674 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.279653 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 09 12:03:21.279860 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.279847 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 09 12:03:21.280185 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.280171 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gfx2l\"" Sep 09 12:03:21.280678 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.280656 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 09 12:03:21.280770 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.280738 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 09 12:03:21.282387 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.282364 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65bb57b978-szsdl"] Sep 09 12:03:21.284002 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.283644 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 09 12:03:21.284107 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.284090 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dlf33seqaqmra\"" Sep 09 12:03:21.319266 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.319245 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-metrics-client-ca\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.319368 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.319278 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-grpc-tls\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.319368 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.319309 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.319368 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.319352 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9688z\" (UniqueName: \"kubernetes.io/projected/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-kube-api-access-9688z\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.319465 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.319383 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.319465 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.319401 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.319465 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.319461 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.319561 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.319501 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-tls\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.420091 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.420047 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.420273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.420118 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-tls\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.420273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.420152 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-metrics-client-ca\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.420273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.420186 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-grpc-tls\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.420273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.420239 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.420273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.420271 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9688z\" (UniqueName: \"kubernetes.io/projected/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-kube-api-access-9688z\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.420539 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.420305 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.420539 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.420333 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.420995 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.420967 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-metrics-client-ca\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.423377 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.423319 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.423377 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.423327 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.424302 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.424265 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.424452 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.424407 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-tls\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.424519 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.424449 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-grpc-tls\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.424701 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.424679 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.433919 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.433329 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9688z\" (UniqueName: \"kubernetes.io/projected/4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b-kube-api-access-9688z\") pod \"thanos-querier-65bb57b978-szsdl\" (UID: \"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b\") " pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:21.586512 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:21.586429 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:22.336887 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.336860 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65bb57b978-szsdl"] Sep 09 12:03:22.355038 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.354955 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 12:03:22.382319 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:22.382291 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7cb4d6_ee19_4df1_9825_5e0846ff1c4b.slice/crio-fe6537da3ff97570b5c67677df4a0907425ad9a088294905eb99004426dbea26 WatchSource:0}: Error finding container fe6537da3ff97570b5c67677df4a0907425ad9a088294905eb99004426dbea26: Status 404 returned error can't find the container with id fe6537da3ff97570b5c67677df4a0907425ad9a088294905eb99004426dbea26 Sep 09 12:03:22.382881 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:22.382850 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cb9d4f_b509_49a6_a74b_837b6d5cb721.slice/crio-684166cb6565e46e8cc07621b275a5067d37aca19fbfed040e444e837e767aad WatchSource:0}: Error finding container 684166cb6565e46e8cc07621b275a5067d37aca19fbfed040e444e837e767aad: Status 404 returned error can't find the container with id 684166cb6565e46e8cc07621b275a5067d37aca19fbfed040e444e837e767aad Sep 09 12:03:22.631584 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.631503 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b99569b85-vhd4m"] Sep 09 12:03:22.633973 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.633947 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.637746 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.637715 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 12:03:22.637908 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.637881 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xschk\"" Sep 09 12:03:22.639155 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.638610 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 09 12:03:22.639155 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.638639 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 09 12:03:22.639155 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.638809 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 09 12:03:22.640962 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.640675 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bakgqvvfnn1th\"" Sep 09 12:03:22.648491 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.648450 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b99569b85-vhd4m"] Sep 09 12:03:22.735026 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.734994 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0244901a-7f18-428d-b6a1-bcfc699f8d10-client-ca-bundle\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.735193 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.735041 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0244901a-7f18-428d-b6a1-bcfc699f8d10-secret-metrics-server-client-certs\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.735193 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.735116 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0244901a-7f18-428d-b6a1-bcfc699f8d10-audit-log\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.735193 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.735162 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0244901a-7f18-428d-b6a1-bcfc699f8d10-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.735336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.735193 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0244901a-7f18-428d-b6a1-bcfc699f8d10-metrics-server-audit-profiles\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.735417 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.735394 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0244901a-7f18-428d-b6a1-bcfc699f8d10-secret-metrics-server-tls\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.735478 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.735453 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8xm\" (UniqueName: \"kubernetes.io/projected/0244901a-7f18-428d-b6a1-bcfc699f8d10-kube-api-access-ml8xm\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.785148 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.785119 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754"] Sep 09 12:03:22.786981 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.786963 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" Sep 09 12:03:22.790282 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.790257 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2pdmj\"" Sep 09 12:03:22.792337 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.792320 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 09 12:03:22.802338 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.802304 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754"] Sep 09 12:03:22.836151 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.836124 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0244901a-7f18-428d-b6a1-bcfc699f8d10-client-ca-bundle\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.836281 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.836157 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0244901a-7f18-428d-b6a1-bcfc699f8d10-secret-metrics-server-client-certs\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.836281 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.836176 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0244901a-7f18-428d-b6a1-bcfc699f8d10-audit-log\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.836281 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.836199 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gfbb\" (UniqueName: \"kubernetes.io/projected/c7f68e41-8cfb-46c0-872d-c4edc20b9032-kube-api-access-6gfbb\") pod \"monitoring-plugin-85bbbbbf4-pt754\" (UID: \"c7f68e41-8cfb-46c0-872d-c4edc20b9032\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" Sep 09 12:03:22.836281 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.836238 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0244901a-7f18-428d-b6a1-bcfc699f8d10-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.836281 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.836265 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0244901a-7f18-428d-b6a1-bcfc699f8d10-metrics-server-audit-profiles\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.836536 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.836381 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0244901a-7f18-428d-b6a1-bcfc699f8d10-secret-metrics-server-tls\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.836536 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.836435 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8xm\" (UniqueName: \"kubernetes.io/projected/0244901a-7f18-428d-b6a1-bcfc699f8d10-kube-api-access-ml8xm\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.836536 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.836467 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c7f68e41-8cfb-46c0-872d-c4edc20b9032-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-pt754\" (UID: \"c7f68e41-8cfb-46c0-872d-c4edc20b9032\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" Sep 09 12:03:22.836679 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.836572 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0244901a-7f18-428d-b6a1-bcfc699f8d10-audit-log\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.837275 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.837222 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0244901a-7f18-428d-b6a1-bcfc699f8d10-metrics-server-audit-profiles\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.837576 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.837553 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0244901a-7f18-428d-b6a1-bcfc699f8d10-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.838832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.838809 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0244901a-7f18-428d-b6a1-bcfc699f8d10-secret-metrics-server-tls\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.838974 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.838961 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0244901a-7f18-428d-b6a1-bcfc699f8d10-secret-metrics-server-client-certs\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.839022 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.839004 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0244901a-7f18-428d-b6a1-bcfc699f8d10-client-ca-bundle\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.845384 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.845365 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8xm\" (UniqueName: \"kubernetes.io/projected/0244901a-7f18-428d-b6a1-bcfc699f8d10-kube-api-access-ml8xm\") pod \"metrics-server-5b99569b85-vhd4m\" (UID: \"0244901a-7f18-428d-b6a1-bcfc699f8d10\") " pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.937259 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.937227 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c7f68e41-8cfb-46c0-872d-c4edc20b9032-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-pt754\" (UID: \"c7f68e41-8cfb-46c0-872d-c4edc20b9032\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" Sep 09 12:03:22.937376 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.937345 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gfbb\" (UniqueName: \"kubernetes.io/projected/c7f68e41-8cfb-46c0-872d-c4edc20b9032-kube-api-access-6gfbb\") pod \"monitoring-plugin-85bbbbbf4-pt754\" (UID: \"c7f68e41-8cfb-46c0-872d-c4edc20b9032\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" Sep 09 12:03:22.937677 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:22.937652 2556 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 09 12:03:22.937766 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:03:22.937732 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f68e41-8cfb-46c0-872d-c4edc20b9032-monitoring-plugin-cert podName:c7f68e41-8cfb-46c0-872d-c4edc20b9032 nodeName:}" failed. No retries permitted until 2026-09-09 12:03:23.437710397 +0000 UTC m=+59.117616579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c7f68e41-8cfb-46c0-872d-c4edc20b9032-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-pt754" (UID: "c7f68e41-8cfb-46c0-872d-c4edc20b9032") : secret "monitoring-plugin-cert" not found Sep 09 12:03:22.947028 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.946827 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:22.950244 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:22.950220 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gfbb\" (UniqueName: \"kubernetes.io/projected/c7f68e41-8cfb-46c0-872d-c4edc20b9032-kube-api-access-6gfbb\") pod \"monitoring-plugin-85bbbbbf4-pt754\" (UID: \"c7f68e41-8cfb-46c0-872d-c4edc20b9032\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" Sep 09 12:03:23.046377 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.046344 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" event={"ID":"5d0a2e6f-20b8-44d8-98cf-9433a0604560","Type":"ContainerStarted","Data":"c9f1e08e27cfad71f8ef7c52f16c7cfc91708c5ba2c4fe8158edf2f4907f779b"} Sep 09 12:03:23.049034 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.048704 2556 generic.go:356] "Generic (PLEG): container finished" podID="52be1a8c-fc1d-40ab-8294-c75bae298228" containerID="696e7af67cb9c26373b244d8fcd7afee174e2e541caad45689ed72d298af84dd" exitCode=0 Sep 09 12:03:23.049034 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.048801 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rxfq" event={"ID":"52be1a8c-fc1d-40ab-8294-c75bae298228","Type":"ContainerDied","Data":"696e7af67cb9c26373b244d8fcd7afee174e2e541caad45689ed72d298af84dd"} Sep 09 12:03:23.052943 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.052916 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" event={"ID":"e41430b3-e44e-4e0c-8b90-ac215547e15f","Type":"ContainerStarted","Data":"7017819b191117de5b265be679a25cfa937be6b3e177d9bbd9438c488f267f54"} Sep 09 12:03:23.053059 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.052976 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" event={"ID":"e41430b3-e44e-4e0c-8b90-ac215547e15f","Type":"ContainerStarted","Data":"19771572e19dc6f674c20a8e08628ba7715696db377acc36e9bcd4fc1adb57e4"} Sep 09 12:03:23.053059 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.053004 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" event={"ID":"e41430b3-e44e-4e0c-8b90-ac215547e15f","Type":"ContainerStarted","Data":"5aecf2d1ce633d82862fa5f95495bcc705f6275ba7d418600955993e1cd849f1"} Sep 09 12:03:23.054732 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.054707 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerStarted","Data":"684166cb6565e46e8cc07621b275a5067d37aca19fbfed040e444e837e767aad"} Sep 09 12:03:23.057039 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.056986 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9blz8" event={"ID":"3a3d2c78-1cae-4271-8944-0437fd466a3b","Type":"ContainerStarted","Data":"cfc3414719b049fd19049128d568187a9180635ddc21cd1c535806a8296c31e8"} Sep 09 12:03:23.058912 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.058868 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dbdb787db-gw7r5" event={"ID":"12287520-b3c1-448b-aed9-5b46fb31a568","Type":"ContainerStarted","Data":"93b9729a98f3b9994f9e25b386e28f5d2b41c6495701211ade78c32176f25260"} Sep 09 12:03:23.060627 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.060570 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" event={"ID":"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b","Type":"ContainerStarted","Data":"fe6537da3ff97570b5c67677df4a0907425ad9a088294905eb99004426dbea26"} Sep 09 12:03:23.080142 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.080088 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gbngg" podStartSLOduration=1.9430123959999999 podStartE2EDuration="5.080066497s" totalImagesPullingTime="3.137054101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:23.079510713 +0000 UTC m=+58.759416905" watchObservedRunningTime="2026-09-09 12:03:23.080066497 +0000 UTC m=+58.759972685" Sep 09 12:03:23.104841 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.104639 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-t7nkg" podStartSLOduration=2.309788058 podStartE2EDuration="5.104616789s" totalImagesPullingTime="2.794828731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:23.103733303 +0000 UTC m=+58.783639491" watchObservedRunningTime="2026-09-09 12:03:23.104616789 +0000 UTC m=+58.784522976" Sep 09 12:03:23.153275 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.153077 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9blz8" podStartSLOduration=1.998211608 podStartE2EDuration="8.153055562s" totalImagesPullingTime="6.154843954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:23.151475052 +0000 UTC m=+58.831381239" watchObservedRunningTime="2026-09-09 12:03:23.153055562 +0000 UTC m=+58.832961754" Sep 09 12:03:23.185704 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.185626 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dbdb787db-gw7r5" podStartSLOduration=2.733827659 podStartE2EDuration="7.185605976s" totalImagesPullingTime="4.451778317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:23.184559841 +0000 UTC m=+58.864466054" watchObservedRunningTime="2026-09-09 12:03:23.185605976 +0000 UTC m=+58.865512163" Sep 09 12:03:23.238145 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.238111 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b99569b85-vhd4m"] Sep 09 12:03:23.245576 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:23.245461 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0244901a_7f18_428d_b6a1_bcfc699f8d10.slice/crio-a782521f16fcdb0525360459ce72fefce965d9659a6989e6ace137fc98948a08 WatchSource:0}: Error finding container a782521f16fcdb0525360459ce72fefce965d9659a6989e6ace137fc98948a08: Status 404 returned error can't find the container with id a782521f16fcdb0525360459ce72fefce965d9659a6989e6ace137fc98948a08 Sep 09 12:03:23.443199 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.443158 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c7f68e41-8cfb-46c0-872d-c4edc20b9032-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-pt754\" (UID: \"c7f68e41-8cfb-46c0-872d-c4edc20b9032\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" Sep 09 12:03:23.445961 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.445914 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c7f68e41-8cfb-46c0-872d-c4edc20b9032-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-pt754\" (UID: \"c7f68e41-8cfb-46c0-872d-c4edc20b9032\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" Sep 09 12:03:23.696101 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.696078 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" Sep 09 12:03:23.948831 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.948755 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lncg4" Sep 09 12:03:23.970926 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:23.970900 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754"] Sep 09 12:03:23.974125 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:23.974098 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f68e41_8cfb_46c0_872d_c4edc20b9032.slice/crio-0b87f8715f1d05fb21dd8a28ca88ec5863a43e6dd8a9072309525df5c18bcd88 WatchSource:0}: Error finding container 0b87f8715f1d05fb21dd8a28ca88ec5863a43e6dd8a9072309525df5c18bcd88: Status 404 returned error can't find the container with id 0b87f8715f1d05fb21dd8a28ca88ec5863a43e6dd8a9072309525df5c18bcd88 Sep 09 12:03:24.066322 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.066277 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rxfq" event={"ID":"52be1a8c-fc1d-40ab-8294-c75bae298228","Type":"ContainerStarted","Data":"91998419e077ae40359d59047f46120b7608ddacca2750539759aabdd31d109c"} Sep 09 12:03:24.066480 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.066341 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rxfq" event={"ID":"52be1a8c-fc1d-40ab-8294-c75bae298228","Type":"ContainerStarted","Data":"73ecafefa9f71f920ae257a39e93850da0168f6abe78841de4a9745028f79782"} Sep 09 12:03:24.068166 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.068135 2556 generic.go:356] "Generic (PLEG): container finished" podID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerID="99366df7f7d25e6df26a6bc6045b2ab1878ac0b4b54d68ead73bfb52988429bb" exitCode=0 Sep 09 12:03:24.068299 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.068240 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerDied","Data":"99366df7f7d25e6df26a6bc6045b2ab1878ac0b4b54d68ead73bfb52988429bb"} Sep 09 12:03:24.069790 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.069762 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" event={"ID":"0244901a-7f18-428d-b6a1-bcfc699f8d10","Type":"ContainerStarted","Data":"a782521f16fcdb0525360459ce72fefce965d9659a6989e6ace137fc98948a08"} Sep 09 12:03:24.071218 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.071175 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" event={"ID":"c7f68e41-8cfb-46c0-872d-c4edc20b9032","Type":"ContainerStarted","Data":"0b87f8715f1d05fb21dd8a28ca88ec5863a43e6dd8a9072309525df5c18bcd88"} Sep 09 12:03:24.094605 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.094561 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9rxfq" podStartSLOduration=2.323171118 podStartE2EDuration="6.094544805s" totalImagesPullingTime="3.692789944s" totalInitContainerRuntime="78.583743ms" isStatefulPod=true podCreationTimestamp="2026-09-09 12:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:24.094404695 +0000 UTC m=+59.774310884" watchObservedRunningTime="2026-09-09 12:03:24.094544805 +0000 UTC m=+59.774450992" Sep 09 12:03:24.426253 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.426222 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 12:03:24.429865 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.429844 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.433714 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.433694 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 12:03:24.434369 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.434344 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 12:03:24.434568 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.434548 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 12:03:24.434671 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.434652 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f99k4nrn23j15\"" Sep 09 12:03:24.434716 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.434689 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 12:03:24.434856 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.434842 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 12:03:24.435011 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.434995 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 12:03:24.436012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.435270 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 12:03:24.436012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.435280 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2prtd\"" Sep 09 12:03:24.436012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.435546 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 12:03:24.436012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.435770 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 12:03:24.436012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.435866 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 12:03:24.436338 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.436021 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 12:03:24.436393 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.436338 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 12:03:24.437037 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.437017 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 12:03:24.440978 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.440960 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 12:03:24.473903 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.473875 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 12:03:24.557212 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557161 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557398 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557229 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config-out\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557398 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557306 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdx2z\" (UniqueName: \"kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-kube-api-access-xdx2z\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557398 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557343 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557597 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557401 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557597 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557439 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557597 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557465 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557597 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557491 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557597 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557514 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-web-config\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557597 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557551 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557818 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557614 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557818 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557649 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557818 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557709 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557818 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557735 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557818 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557755 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557818 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557770 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557818 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557794 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.557818 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557811 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.558125 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557855 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.558125 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.557906 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.658770 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.658727 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.658943 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.658796 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.658943 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.658829 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config-out\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.658943 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.658855 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdx2z\" (UniqueName: \"kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-kube-api-access-xdx2z\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.658943 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.658884 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.658943 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.658926 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.658954 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.658978 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659004 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659025 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-web-config\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659051 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659113 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659134 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659167 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659183 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659222 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659671 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659239 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659671 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659275 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659671 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659300 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.659671 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.659327 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.660055 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.660024 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.661408 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.660471 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.661408 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.661063 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.662889 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.662859 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.662889 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.662879 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.663048 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.663013 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.663682 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.663591 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config-out\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.663682 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.663608 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-web-config\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.664443 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.664373 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.664443 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.664384 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.664443 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.664385 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.664715 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.664664 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.664805 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.664726 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 12:03:24.664906 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.664727 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 12:03:24.665542 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.665522 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.665856 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.665837 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.666631 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.666612 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 12:03:24.670284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.670265 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 12:03:24.671481 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.671459 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 12:03:24.672424 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.672401 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.672503 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.672411 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.674677 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.674653 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.675148 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.675130 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.675562 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.675543 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.677004 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.676951 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdx2z\" (UniqueName: \"kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-kube-api-access-xdx2z\") pod \"prometheus-k8s-0\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:24.689023 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.688996 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dbdb787db-gw7r5"] Sep 09 12:03:24.747685 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.747658 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2prtd\"" Sep 09 12:03:24.753212 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:24.753179 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:25.499861 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:25.499791 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 12:03:25.585492 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:25.585450 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ccb7e8_afc4_470c_bfea_ba32b3e204ff.slice/crio-10f0942d8e96d5206515a7cc884ee472e006ac5eabe77531c215fd7089d55507 WatchSource:0}: Error finding container 10f0942d8e96d5206515a7cc884ee472e006ac5eabe77531c215fd7089d55507: Status 404 returned error can't find the container with id 10f0942d8e96d5206515a7cc884ee472e006ac5eabe77531c215fd7089d55507 Sep 09 12:03:26.081543 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.081388 2556 generic.go:356] "Generic (PLEG): container finished" podID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerID="8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0" exitCode=0 Sep 09 12:03:26.081543 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.081435 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerDied","Data":"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0"} Sep 09 12:03:26.081543 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.081508 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerStarted","Data":"10f0942d8e96d5206515a7cc884ee472e006ac5eabe77531c215fd7089d55507"} Sep 09 12:03:26.084622 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.084489 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" event={"ID":"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b","Type":"ContainerStarted","Data":"70ff994f4f64b11eaa1e376bce1791b4d00c49836d53be860d7900ee30bf25a8"} Sep 09 12:03:26.084622 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.084565 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" event={"ID":"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b","Type":"ContainerStarted","Data":"e9f5535dac56251733fabf057844f2f85bd8555b7509e8a3dc25712d0d25acac"} Sep 09 12:03:26.086832 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.086331 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" event={"ID":"0244901a-7f18-428d-b6a1-bcfc699f8d10","Type":"ContainerStarted","Data":"aedb89de51a9a33bdd809bfe47812ee90fb9745733996d7e319e3f03160d19bf"} Sep 09 12:03:26.088620 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.088596 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" event={"ID":"c7f68e41-8cfb-46c0-872d-c4edc20b9032","Type":"ContainerStarted","Data":"0d05782289c4ae65eb7b6d788b46eee6669d4c54d3927211b09a0a4f6ce545a6"} Sep 09 12:03:26.088800 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.088785 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" Sep 09 12:03:26.094645 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.094615 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" Sep 09 12:03:26.143407 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.143339 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-pt754" podStartSLOduration=2.467740478 podStartE2EDuration="4.143318989s" totalImagesPullingTime="1.675578511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:26.141751095 +0000 UTC m=+61.821657283" watchObservedRunningTime="2026-09-09 12:03:26.143318989 +0000 UTC m=+61.823225170" Sep 09 12:03:26.163823 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.163772 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" podStartSLOduration=2.201757454 podStartE2EDuration="4.163755499s" totalImagesPullingTime="1.961998045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:26.163117419 +0000 UTC m=+61.843023619" watchObservedRunningTime="2026-09-09 12:03:26.163755499 +0000 UTC m=+61.843661745" Sep 09 12:03:26.999143 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:26.999106 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:27.096760 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:27.096728 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerStarted","Data":"f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71"} Sep 09 12:03:27.096904 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:27.096815 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerStarted","Data":"8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824"} Sep 09 12:03:27.096904 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:27.096862 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerStarted","Data":"7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c"} Sep 09 12:03:27.096904 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:27.096901 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerStarted","Data":"b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906"} Sep 09 12:03:27.097037 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:27.096939 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerStarted","Data":"c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5"} Sep 09 12:03:27.099559 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:27.099533 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" event={"ID":"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b","Type":"ContainerStarted","Data":"c7b04ce1b9cd856f788b0c20abe18402b46ddb4d625ca698de44d2a32bdea205"} Sep 09 12:03:28.107284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:28.107245 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" event={"ID":"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b","Type":"ContainerStarted","Data":"5bb8717e0313696d8991d97eed981ef3642eb1d41c89f22c6dce049378961e87"} Sep 09 12:03:28.107744 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:28.107342 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" event={"ID":"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b","Type":"ContainerStarted","Data":"5532d9a27a58ee21964fd45c66fba46e989f94af0ad5b7e97dea37ec69bcf840"} Sep 09 12:03:28.107744 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:28.107395 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" event={"ID":"4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b","Type":"ContainerStarted","Data":"f1bd9d84758c0075a116a32c5433abd843c0eb6b022696ab93ff1eb3818b7621"} Sep 09 12:03:29.031115 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:29.031076 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r564d" Sep 09 12:03:29.052824 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:29.052775 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" podStartSLOduration=4.253068669 podStartE2EDuration="8.052758769s" totalImagesPullingTime="3.7996901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:03:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:28.151317481 +0000 UTC m=+63.831223668" watchObservedRunningTime="2026-09-09 12:03:29.052758769 +0000 UTC m=+64.732664956" Sep 09 12:03:29.112290 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:29.112259 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:30.118777 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.118740 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerStarted","Data":"dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80"} Sep 09 12:03:30.122155 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.122133 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerStarted","Data":"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49"} Sep 09 12:03:30.122295 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.122185 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerStarted","Data":"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c"} Sep 09 12:03:30.122295 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.122227 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerStarted","Data":"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c"} Sep 09 12:03:30.122295 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.122255 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerStarted","Data":"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9"} Sep 09 12:03:30.122295 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.122280 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerStarted","Data":"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695"} Sep 09 12:03:30.122416 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.122304 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerStarted","Data":"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7"} Sep 09 12:03:30.128790 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.128770 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-65bb57b978-szsdl" Sep 09 12:03:30.156425 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.156373 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.049621926 podStartE2EDuration="11.156356428s" totalImagesPullingTime="6.086087255s" totalInitContainerRuntime="20.647247ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:03:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:30.153917841 +0000 UTC m=+65.833824029" watchObservedRunningTime="2026-09-09 12:03:30.156356428 +0000 UTC m=+65.836262614" Sep 09 12:03:30.195696 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.195639 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.115358012 podStartE2EDuration="6.195621624s" totalImagesPullingTime="3.040500318s" totalInitContainerRuntime="39.763294ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:30.193362167 +0000 UTC m=+65.873268355" watchObservedRunningTime="2026-09-09 12:03:30.195621624 +0000 UTC m=+65.875527810" Sep 09 12:03:30.620785 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.620746 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:30.620983 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.620807 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:30.623935 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.623906 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 12:03:30.624039 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.623913 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 12:03:30.633637 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.633613 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/556aa98a-d697-4e96-beda-0ae8d694b31d-original-pull-secret\") pod \"global-pull-secret-syncer-kzvf5\" (UID: \"556aa98a-d697-4e96-beda-0ae8d694b31d\") " pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:30.633745 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.633640 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b400cd-b876-4e32-b6c1-a1322eb20817-metrics-certs\") pod \"network-metrics-daemon-rktvs\" (UID: \"40b400cd-b876-4e32-b6c1-a1322eb20817\") " pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:30.643896 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.643875 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-85k95\"" Sep 09 12:03:30.651665 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.651637 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rktvs" Sep 09 12:03:30.721866 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.721832 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pw99\" (UniqueName: \"kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99\") pod \"network-check-target-b67ss\" (UID: \"79debca6-ea95-4e3d-8952-5cf5ed2ab970\") " pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:30.725372 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.725343 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 12:03:30.735792 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.735763 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 12:03:30.745353 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.745334 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pw99\" (UniqueName: \"kubernetes.io/projected/79debca6-ea95-4e3d-8952-5cf5ed2ab970-kube-api-access-8pw99\") pod \"network-check-target-b67ss\" (UID: \"79debca6-ea95-4e3d-8952-5cf5ed2ab970\") " pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:30.926816 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.926785 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzvf5" Sep 09 12:03:30.928921 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.928900 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rktvs"] Sep 09 12:03:30.930749 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:30.930723 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b400cd_b876_4e32_b6c1_a1322eb20817.slice/crio-3b4b78ee266372560d3be230032eab7c041a18aed815fc65d5168576630227c5 WatchSource:0}: Error finding container 3b4b78ee266372560d3be230032eab7c041a18aed815fc65d5168576630227c5: Status 404 returned error can't find the container with id 3b4b78ee266372560d3be230032eab7c041a18aed815fc65d5168576630227c5 Sep 09 12:03:30.940266 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.940239 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8phgx\"" Sep 09 12:03:30.945547 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:30.945525 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:31.131316 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:31.131269 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rktvs" event={"ID":"40b400cd-b876-4e32-b6c1-a1322eb20817","Type":"ContainerStarted","Data":"3b4b78ee266372560d3be230032eab7c041a18aed815fc65d5168576630227c5"} Sep 09 12:03:31.204003 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:31.203977 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kzvf5"] Sep 09 12:03:31.206192 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:31.206159 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556aa98a_d697_4e96_beda_0ae8d694b31d.slice/crio-9bfadfebccc5cd9c0348ac10c62bfc51857acf8bc8205c75e7dfad8a7755f907 WatchSource:0}: Error finding container 9bfadfebccc5cd9c0348ac10c62bfc51857acf8bc8205c75e7dfad8a7755f907: Status 404 returned error can't find the container with id 9bfadfebccc5cd9c0348ac10c62bfc51857acf8bc8205c75e7dfad8a7755f907 Sep 09 12:03:31.228745 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:31.228705 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b67ss"] Sep 09 12:03:31.231955 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:03:31.231930 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79debca6_ea95_4e3d_8952_5cf5ed2ab970.slice/crio-e64d00da716a4e89d6566e45d4d35a52fb733b2cbb84e20ea68064685f6497ea WatchSource:0}: Error finding container e64d00da716a4e89d6566e45d4d35a52fb733b2cbb84e20ea68064685f6497ea: Status 404 returned error can't find the container with id e64d00da716a4e89d6566e45d4d35a52fb733b2cbb84e20ea68064685f6497ea Sep 09 12:03:32.141410 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:32.141363 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b67ss" event={"ID":"79debca6-ea95-4e3d-8952-5cf5ed2ab970","Type":"ContainerStarted","Data":"e64d00da716a4e89d6566e45d4d35a52fb733b2cbb84e20ea68064685f6497ea"} Sep 09 12:03:32.143036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:32.142993 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kzvf5" event={"ID":"556aa98a-d697-4e96-beda-0ae8d694b31d","Type":"ContainerStarted","Data":"9bfadfebccc5cd9c0348ac10c62bfc51857acf8bc8205c75e7dfad8a7755f907"} Sep 09 12:03:34.158888 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:34.158847 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rktvs" event={"ID":"40b400cd-b876-4e32-b6c1-a1322eb20817","Type":"ContainerStarted","Data":"ab9640d031b04f5e14db571d78b344f9ada08bd6027d9b5392bedbfed4d53c6c"} Sep 09 12:03:34.159365 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:34.158913 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rktvs" event={"ID":"40b400cd-b876-4e32-b6c1-a1322eb20817","Type":"ContainerStarted","Data":"29c200045ed601a3a9ed8b0545474172ccf63c6ff0a02f531f01d6b25e68770a"} Sep 09 12:03:34.179300 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:34.179248 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rktvs" podStartSLOduration=67.002950641 podStartE2EDuration="1m9.179232177s" totalImagesPullingTime="2.176281536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:34.179066603 +0000 UTC m=+69.858972801" watchObservedRunningTime="2026-09-09 12:03:34.179232177 +0000 UTC m=+69.859138361" Sep 09 12:03:34.753592 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:34.753555 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:03:37.173937 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:37.173897 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kzvf5" event={"ID":"556aa98a-d697-4e96-beda-0ae8d694b31d","Type":"ContainerStarted","Data":"106290f722814f7b72d87f50fcc9868ac56371f2beb53da451c60dee7f30cbc4"} Sep 09 12:03:37.175458 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:37.175435 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b67ss" event={"ID":"79debca6-ea95-4e3d-8952-5cf5ed2ab970","Type":"ContainerStarted","Data":"d0f626b1e002f750be3c72fdeb453cd82dc7d1660c16745a0ed7fe47637ec900"} Sep 09 12:03:37.175584 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:37.175572 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:03:37.194246 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:37.194176 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kzvf5" podStartSLOduration=67.097782855 podStartE2EDuration="1m12.1941619s" totalImagesPullingTime="5.096379045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:37.193613199 +0000 UTC m=+72.873519389" watchObservedRunningTime="2026-09-09 12:03:37.1941619 +0000 UTC m=+72.874068087" Sep 09 12:03:37.217641 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:37.217601 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b67ss" podStartSLOduration=67.151903961 podStartE2EDuration="1m12.217588047s" totalImagesPullingTime="5.065684086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:03:37.216699408 +0000 UTC m=+72.896605596" watchObservedRunningTime="2026-09-09 12:03:37.217588047 +0000 UTC m=+72.897494233" Sep 09 12:03:42.947937 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:42.947903 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:42.948337 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:42.947956 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:03:50.100100 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.099937 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6dbdb787db-gw7r5" podUID="12287520-b3c1-448b-aed9-5b46fb31a568" containerName="console" containerID="cri-o://93b9729a98f3b9994f9e25b386e28f5d2b41c6495701211ade78c32176f25260" gracePeriod=15 Sep 09 12:03:50.227011 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.226979 2556 generic.go:356] "Generic (PLEG): container finished" podID="12287520-b3c1-448b-aed9-5b46fb31a568" containerID="93b9729a98f3b9994f9e25b386e28f5d2b41c6495701211ade78c32176f25260" exitCode=0 Sep 09 12:03:50.227143 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.227014 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dbdb787db-gw7r5" event={"ID":"12287520-b3c1-448b-aed9-5b46fb31a568","Type":"ContainerDied","Data":"93b9729a98f3b9994f9e25b386e28f5d2b41c6495701211ade78c32176f25260"} Sep 09 12:03:50.335605 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.335582 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:50.402823 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.402735 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-console-config\") pod \"12287520-b3c1-448b-aed9-5b46fb31a568\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " Sep 09 12:03:50.402823 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.402782 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d7vb\" (UniqueName: \"kubernetes.io/projected/12287520-b3c1-448b-aed9-5b46fb31a568-kube-api-access-9d7vb\") pod \"12287520-b3c1-448b-aed9-5b46fb31a568\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " Sep 09 12:03:50.402823 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.402800 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-oauth-config\") pod \"12287520-b3c1-448b-aed9-5b46fb31a568\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " Sep 09 12:03:50.402823 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.402817 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-oauth-serving-cert\") pod \"12287520-b3c1-448b-aed9-5b46fb31a568\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " Sep 09 12:03:50.403136 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.402862 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-trusted-ca-bundle\") pod \"12287520-b3c1-448b-aed9-5b46fb31a568\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " Sep 09 12:03:50.403136 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.402886 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-serving-cert\") pod \"12287520-b3c1-448b-aed9-5b46fb31a568\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " Sep 09 12:03:50.403136 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.402948 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-service-ca\") pod \"12287520-b3c1-448b-aed9-5b46fb31a568\" (UID: \"12287520-b3c1-448b-aed9-5b46fb31a568\") " Sep 09 12:03:50.403331 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.403156 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-console-config" (OuterVolumeSpecName: "console-config") pod "12287520-b3c1-448b-aed9-5b46fb31a568" (UID: "12287520-b3c1-448b-aed9-5b46fb31a568"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:03:50.403424 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.403392 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "12287520-b3c1-448b-aed9-5b46fb31a568" (UID: "12287520-b3c1-448b-aed9-5b46fb31a568"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:03:50.403478 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.403418 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "12287520-b3c1-448b-aed9-5b46fb31a568" (UID: "12287520-b3c1-448b-aed9-5b46fb31a568"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:03:50.403478 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.403450 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-service-ca" (OuterVolumeSpecName: "service-ca") pod "12287520-b3c1-448b-aed9-5b46fb31a568" (UID: "12287520-b3c1-448b-aed9-5b46fb31a568"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:03:50.405165 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.405142 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12287520-b3c1-448b-aed9-5b46fb31a568-kube-api-access-9d7vb" (OuterVolumeSpecName: "kube-api-access-9d7vb") pod "12287520-b3c1-448b-aed9-5b46fb31a568" (UID: "12287520-b3c1-448b-aed9-5b46fb31a568"). InnerVolumeSpecName "kube-api-access-9d7vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:03:50.405492 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.405463 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "12287520-b3c1-448b-aed9-5b46fb31a568" (UID: "12287520-b3c1-448b-aed9-5b46fb31a568"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:03:50.405582 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.405505 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "12287520-b3c1-448b-aed9-5b46fb31a568" (UID: "12287520-b3c1-448b-aed9-5b46fb31a568"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:03:50.503964 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.503927 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:03:50.503964 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.503957 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-console-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:03:50.503964 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.503967 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d7vb\" (UniqueName: \"kubernetes.io/projected/12287520-b3c1-448b-aed9-5b46fb31a568-kube-api-access-9d7vb\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:03:50.504170 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.503977 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-oauth-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:03:50.504170 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.503985 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-oauth-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:03:50.504170 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.503993 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12287520-b3c1-448b-aed9-5b46fb31a568-trusted-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:03:50.504170 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:50.504001 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12287520-b3c1-448b-aed9-5b46fb31a568-console-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:03:51.232388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:51.232351 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dbdb787db-gw7r5" event={"ID":"12287520-b3c1-448b-aed9-5b46fb31a568","Type":"ContainerDied","Data":"977402f7fe0633e5a3d930238f7dedbf45607083024b3b6203a440e63162b74a"} Sep 09 12:03:51.232789 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:51.232404 2556 scope.go:117] "RemoveContainer" containerID="93b9729a98f3b9994f9e25b386e28f5d2b41c6495701211ade78c32176f25260" Sep 09 12:03:51.232789 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:51.232367 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dbdb787db-gw7r5" Sep 09 12:03:51.260673 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:51.260641 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dbdb787db-gw7r5"] Sep 09 12:03:51.270553 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:51.270529 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dbdb787db-gw7r5"] Sep 09 12:03:52.816587 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:03:52.816556 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12287520-b3c1-448b-aed9-5b46fb31a568" path="/var/lib/kubelet/pods/12287520-b3c1-448b-aed9-5b46fb31a568/volumes" Sep 09 12:04:02.953164 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:02.953134 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:04:02.956940 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:02.956918 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b99569b85-vhd4m" Sep 09 12:04:08.182004 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:08.181966 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b67ss" Sep 09 12:04:24.753697 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:24.753661 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:24.772522 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:24.772495 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:25.388033 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:25.388004 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:38.805671 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:38.805635 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 12:04:38.806050 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:38.805914 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="alertmanager" containerID="cri-o://c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5" gracePeriod=120 Sep 09 12:04:38.806050 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:38.805987 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy-web" containerID="cri-o://7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c" gracePeriod=120 Sep 09 12:04:38.806050 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:38.805988 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy-metric" containerID="cri-o://f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71" gracePeriod=120 Sep 09 12:04:38.806284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:38.806045 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy" containerID="cri-o://8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824" gracePeriod=120 Sep 09 12:04:38.806284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:38.806064 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="prom-label-proxy" containerID="cri-o://dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80" gracePeriod=120 Sep 09 12:04:38.806284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:38.806110 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="config-reloader" containerID="cri-o://b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906" gracePeriod=120 Sep 09 12:04:39.427251 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:39.427199 2556 generic.go:356] "Generic (PLEG): container finished" podID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerID="dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80" exitCode=0 Sep 09 12:04:39.427251 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:39.427243 2556 generic.go:356] "Generic (PLEG): container finished" podID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerID="f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71" exitCode=0 Sep 09 12:04:39.427251 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:39.427252 2556 generic.go:356] "Generic (PLEG): container finished" podID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerID="8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824" exitCode=0 Sep 09 12:04:39.427251 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:39.427258 2556 generic.go:356] "Generic (PLEG): container finished" podID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerID="b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906" exitCode=0 Sep 09 12:04:39.427497 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:39.427266 2556 generic.go:356] "Generic (PLEG): container finished" podID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerID="c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5" exitCode=0 Sep 09 12:04:39.427497 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:39.427273 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerDied","Data":"dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80"} Sep 09 12:04:39.427497 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:39.427337 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerDied","Data":"f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71"} Sep 09 12:04:39.427497 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:39.427364 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerDied","Data":"8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824"} Sep 09 12:04:39.427497 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:39.427388 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerDied","Data":"b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906"} Sep 09 12:04:39.427497 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:39.427414 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerDied","Data":"c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5"} Sep 09 12:04:40.068284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.068265 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.186727 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.186696 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-main-tls\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.186874 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.186735 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-cluster-tls-config\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.186874 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.186753 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.186874 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.186794 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgwpr\" (UniqueName: \"kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-kube-api-access-bgwpr\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.186874 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.186815 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.186874 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.186839 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-web-config\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.186874 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.186864 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-out\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.187243 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.187220 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-web\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.187311 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.187292 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-volume\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.187371 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.187321 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-trusted-ca-bundle\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.187371 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.187359 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-metrics-client-ca\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.187470 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.187390 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-tls-assets\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.187470 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.187417 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-main-db\") pod \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\" (UID: \"b6cb9d4f-b509-49a6-a74b-837b6d5cb721\") " Sep 09 12:04:40.188300 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.188001 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:04:40.189971 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.189890 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:40.190127 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.190099 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-kube-api-access-bgwpr" (OuterVolumeSpecName: "kube-api-access-bgwpr") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "kube-api-access-bgwpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:04:40.190338 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.190318 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:04:40.190405 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.190365 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:04:40.190747 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.190643 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:40.190747 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.190658 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-out" (OuterVolumeSpecName: "config-out") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:04:40.190877 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.190847 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:40.190984 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.190961 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:40.192033 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.192013 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:04:40.192407 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.192381 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-volume" (OuterVolumeSpecName: "config-volume") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:40.193891 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.193867 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:40.200239 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.200195 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-web-config" (OuterVolumeSpecName: "web-config") pod "b6cb9d4f-b509-49a6-a74b-837b6d5cb721" (UID: "b6cb9d4f-b509-49a6-a74b-837b6d5cb721"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:40.288700 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288654 2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-tls-assets\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288700 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288692 2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-main-db\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288700 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288703 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-main-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288700 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288713 2556 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-cluster-tls-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288934 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288724 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288934 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288733 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgwpr\" (UniqueName: \"kubernetes.io/projected/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-kube-api-access-bgwpr\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288934 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288743 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288934 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288752 2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-web-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288934 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288760 2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-out\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288934 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288767 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288934 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288775 2556 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288934 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288783 2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.288934 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.288791 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb9d4f-b509-49a6-a74b-837b6d5cb721-metrics-client-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:40.435024 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.434988 2556 generic.go:356] "Generic (PLEG): container finished" podID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerID="7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c" exitCode=0 Sep 09 12:04:40.435197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.435066 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.435197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.435064 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerDied","Data":"7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c"} Sep 09 12:04:40.435323 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.435216 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6cb9d4f-b509-49a6-a74b-837b6d5cb721","Type":"ContainerDied","Data":"684166cb6565e46e8cc07621b275a5067d37aca19fbfed040e444e837e767aad"} Sep 09 12:04:40.435323 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.435227 2556 scope.go:117] "RemoveContainer" containerID="dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80" Sep 09 12:04:40.443576 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.443547 2556 scope.go:117] "RemoveContainer" containerID="f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71" Sep 09 12:04:40.450478 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.450458 2556 scope.go:117] "RemoveContainer" containerID="8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824" Sep 09 12:04:40.456695 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.456675 2556 scope.go:117] "RemoveContainer" containerID="7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c" Sep 09 12:04:40.463027 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.463010 2556 scope.go:117] "RemoveContainer" containerID="b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906" Sep 09 12:04:40.469522 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.469502 2556 scope.go:117] "RemoveContainer" containerID="c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5" Sep 09 12:04:40.474172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.474145 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 12:04:40.481515 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.481495 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 12:04:40.483302 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.483287 2556 scope.go:117] "RemoveContainer" containerID="99366df7f7d25e6df26a6bc6045b2ab1878ac0b4b54d68ead73bfb52988429bb" Sep 09 12:04:40.489987 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.489968 2556 scope.go:117] "RemoveContainer" containerID="dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80" Sep 09 12:04:40.490303 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:40.490269 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80\": container with ID starting with dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80 not found: ID does not exist" containerID="dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80" Sep 09 12:04:40.490363 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.490313 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80"} err="failed to get container status \"dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80\": rpc error: code = NotFound desc = could not find container \"dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80\": container with ID starting with dce41753c3995e478d6f72e7fbe710d0896388720bad5b846dc69648948d1c80 not found: ID does not exist" Sep 09 12:04:40.490363 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.490355 2556 scope.go:117] "RemoveContainer" containerID="f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71" Sep 09 12:04:40.490659 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:40.490642 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71\": container with ID starting with f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71 not found: ID does not exist" containerID="f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71" Sep 09 12:04:40.490714 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.490663 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71"} err="failed to get container status \"f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71\": rpc error: code = NotFound desc = could not find container \"f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71\": container with ID starting with f6f7677dbb95bb1d8e11b689fdbecbc1572f25b16cec7d20b8a42004bea68f71 not found: ID does not exist" Sep 09 12:04:40.490714 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.490683 2556 scope.go:117] "RemoveContainer" containerID="8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824" Sep 09 12:04:40.490908 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:40.490894 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824\": container with ID starting with 8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824 not found: ID does not exist" containerID="8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824" Sep 09 12:04:40.490947 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.490913 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824"} err="failed to get container status \"8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824\": rpc error: code = NotFound desc = could not find container \"8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824\": container with ID starting with 8736cd0213c2c28f072041a3debc8763cd58a84e9b44a65c4887b9d894c1f824 not found: ID does not exist" Sep 09 12:04:40.490947 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.490929 2556 scope.go:117] "RemoveContainer" containerID="7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c" Sep 09 12:04:40.491167 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:40.491149 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c\": container with ID starting with 7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c not found: ID does not exist" containerID="7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c" Sep 09 12:04:40.491243 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.491175 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c"} err="failed to get container status \"7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c\": rpc error: code = NotFound desc = could not find container \"7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c\": container with ID starting with 7ca85bf39f42af7920782ecc81a33631c8e075f87fe1c892cf85736c11d9881c not found: ID does not exist" Sep 09 12:04:40.491243 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.491192 2556 scope.go:117] "RemoveContainer" containerID="b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906" Sep 09 12:04:40.491527 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:40.491513 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906\": container with ID starting with b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906 not found: ID does not exist" containerID="b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906" Sep 09 12:04:40.491578 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.491530 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906"} err="failed to get container status \"b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906\": rpc error: code = NotFound desc = could not find container \"b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906\": container with ID starting with b43274c8a49fb39ff29d8808ce0818727f8739e63952d95706212a3e991c4906 not found: ID does not exist" Sep 09 12:04:40.491578 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.491541 2556 scope.go:117] "RemoveContainer" containerID="c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5" Sep 09 12:04:40.491748 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:40.491733 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5\": container with ID starting with c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5 not found: ID does not exist" containerID="c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5" Sep 09 12:04:40.491787 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.491750 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5"} err="failed to get container status \"c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5\": rpc error: code = NotFound desc = could not find container \"c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5\": container with ID starting with c6acce0c785a72b54981b18ab9b7ef7023e0c349d296a1168c36974c4b6f1dd5 not found: ID does not exist" Sep 09 12:04:40.491787 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.491762 2556 scope.go:117] "RemoveContainer" containerID="99366df7f7d25e6df26a6bc6045b2ab1878ac0b4b54d68ead73bfb52988429bb" Sep 09 12:04:40.491985 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:40.491971 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99366df7f7d25e6df26a6bc6045b2ab1878ac0b4b54d68ead73bfb52988429bb\": container with ID starting with 99366df7f7d25e6df26a6bc6045b2ab1878ac0b4b54d68ead73bfb52988429bb not found: ID does not exist" containerID="99366df7f7d25e6df26a6bc6045b2ab1878ac0b4b54d68ead73bfb52988429bb" Sep 09 12:04:40.492031 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.491987 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99366df7f7d25e6df26a6bc6045b2ab1878ac0b4b54d68ead73bfb52988429bb"} err="failed to get container status \"99366df7f7d25e6df26a6bc6045b2ab1878ac0b4b54d68ead73bfb52988429bb\": rpc error: code = NotFound desc = could not find container \"99366df7f7d25e6df26a6bc6045b2ab1878ac0b4b54d68ead73bfb52988429bb\": container with ID starting with 99366df7f7d25e6df26a6bc6045b2ab1878ac0b4b54d68ead73bfb52988429bb not found: ID does not exist" Sep 09 12:04:40.526865 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.526839 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 12:04:40.527131 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527119 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy-metric" Sep 09 12:04:40.527169 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527133 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy-metric" Sep 09 12:04:40.527169 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527141 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="prom-label-proxy" Sep 09 12:04:40.527169 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527146 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="prom-label-proxy" Sep 09 12:04:40.527169 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527158 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="alertmanager" Sep 09 12:04:40.527169 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527163 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="alertmanager" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527170 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy-web" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527176 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy-web" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527184 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527190 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527198 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12287520-b3c1-448b-aed9-5b46fb31a568" containerName="console" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527217 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="12287520-b3c1-448b-aed9-5b46fb31a568" containerName="console" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527223 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="config-reloader" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527229 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="config-reloader" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527237 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="init-config-reloader" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527243 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="init-config-reloader" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527285 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy-web" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527293 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527300 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="alertmanager" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527305 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="config-reloader" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527312 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="12287520-b3c1-448b-aed9-5b46fb31a568" containerName="console" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527318 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="kube-rbac-proxy-metric" Sep 09 12:04:40.527336 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.527322 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" containerName="prom-label-proxy" Sep 09 12:04:40.530867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.530851 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.534991 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.534973 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 12:04:40.535126 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.535106 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 12:04:40.535255 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.535239 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t8bvx\"" Sep 09 12:04:40.535949 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.535919 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 12:04:40.536044 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.535929 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 12:04:40.536044 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.535992 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 12:04:40.536044 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.536039 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 12:04:40.536245 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.536040 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 12:04:40.536467 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.536448 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 12:04:40.545188 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.545168 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 12:04:40.559615 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.559595 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 12:04:40.691912 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.691885 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692102 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.691920 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/98045cb3-41cc-4912-95d0-8b4c30178891-tls-assets\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692102 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.691937 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98045cb3-41cc-4912-95d0-8b4c30178891-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692102 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.691963 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98045cb3-41cc-4912-95d0-8b4c30178891-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692102 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.692050 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/98045cb3-41cc-4912-95d0-8b4c30178891-config-out\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692102 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.692082 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ms8\" (UniqueName: \"kubernetes.io/projected/98045cb3-41cc-4912-95d0-8b4c30178891-kube-api-access-b4ms8\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692408 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.692125 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692408 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.692148 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692408 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.692180 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692408 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.692243 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/98045cb3-41cc-4912-95d0-8b4c30178891-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692408 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.692298 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692408 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.692328 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-config-volume\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.692408 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.692351 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-web-config\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793137 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-config-volume\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793170 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-web-config\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793194 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793234 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/98045cb3-41cc-4912-95d0-8b4c30178891-tls-assets\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793249 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98045cb3-41cc-4912-95d0-8b4c30178891-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793267 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98045cb3-41cc-4912-95d0-8b4c30178891-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793290 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/98045cb3-41cc-4912-95d0-8b4c30178891-config-out\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793304 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ms8\" (UniqueName: \"kubernetes.io/projected/98045cb3-41cc-4912-95d0-8b4c30178891-kube-api-access-b4ms8\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793331 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793353 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793382 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793417 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/98045cb3-41cc-4912-95d0-8b4c30178891-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.793489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.793440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.794096 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.794059 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98045cb3-41cc-4912-95d0-8b4c30178891-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.794194 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.794151 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98045cb3-41cc-4912-95d0-8b4c30178891-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.796110 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.796078 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/98045cb3-41cc-4912-95d0-8b4c30178891-config-out\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.796403 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.796347 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.796403 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.796378 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/98045cb3-41cc-4912-95d0-8b4c30178891-tls-assets\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.796541 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.796517 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.796624 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.796601 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-config-volume\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.796686 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.796610 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/98045cb3-41cc-4912-95d0-8b4c30178891-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.796751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.796727 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-web-config\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.796807 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.796776 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.796887 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.796869 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.798080 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.798052 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98045cb3-41cc-4912-95d0-8b4c30178891-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.803109 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.803087 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ms8\" (UniqueName: \"kubernetes.io/projected/98045cb3-41cc-4912-95d0-8b4c30178891-kube-api-access-b4ms8\") pod \"alertmanager-main-0\" (UID: \"98045cb3-41cc-4912-95d0-8b4c30178891\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:40.815536 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.815514 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6cb9d4f-b509-49a6-a74b-837b6d5cb721" path="/var/lib/kubelet/pods/b6cb9d4f-b509-49a6-a74b-837b6d5cb721/volumes" Sep 09 12:04:40.839513 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:40.839497 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 12:04:41.124821 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:41.124792 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 12:04:41.125526 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:04:41.125501 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98045cb3_41cc_4912_95d0_8b4c30178891.slice/crio-49a32889bb370bedd8328a76c48583f03a49f5b96c705e0552fd99943ea480a4 WatchSource:0}: Error finding container 49a32889bb370bedd8328a76c48583f03a49f5b96c705e0552fd99943ea480a4: Status 404 returned error can't find the container with id 49a32889bb370bedd8328a76c48583f03a49f5b96c705e0552fd99943ea480a4 Sep 09 12:04:41.441027 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:41.440997 2556 generic.go:356] "Generic (PLEG): container finished" podID="98045cb3-41cc-4912-95d0-8b4c30178891" containerID="5a8c631f321bff84c9772ec50d1b03a236c52bb796675ed5a0e5cfba90a4f4bb" exitCode=0 Sep 09 12:04:41.441239 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:41.441070 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98045cb3-41cc-4912-95d0-8b4c30178891","Type":"ContainerDied","Data":"5a8c631f321bff84c9772ec50d1b03a236c52bb796675ed5a0e5cfba90a4f4bb"} Sep 09 12:04:41.441239 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:41.441139 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98045cb3-41cc-4912-95d0-8b4c30178891","Type":"ContainerStarted","Data":"49a32889bb370bedd8328a76c48583f03a49f5b96c705e0552fd99943ea480a4"} Sep 09 12:04:42.447628 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.447594 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98045cb3-41cc-4912-95d0-8b4c30178891","Type":"ContainerStarted","Data":"f92fbe2d433704f263ae6d39f1e754f84a7b1dec7ae1a624db60da3b1ff981a5"} Sep 09 12:04:42.447973 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.447657 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98045cb3-41cc-4912-95d0-8b4c30178891","Type":"ContainerStarted","Data":"b23e250f75864b9e216dbdabe1c5e4a8d247d862236473cf93bf4439203174bc"} Sep 09 12:04:42.447973 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.447686 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98045cb3-41cc-4912-95d0-8b4c30178891","Type":"ContainerStarted","Data":"4f7b3852edac2f87a4cac466b4e2c0b54ccf988a4d0dcc5bcc6f9c1606db21bb"} Sep 09 12:04:42.447973 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.447710 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98045cb3-41cc-4912-95d0-8b4c30178891","Type":"ContainerStarted","Data":"e60287e8a9f33dcc604225b5d2a0683d78db6c2529083f59d81d63946888b44f"} Sep 09 12:04:42.447973 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.447732 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98045cb3-41cc-4912-95d0-8b4c30178891","Type":"ContainerStarted","Data":"80c4daeef7391f13ee05e286b3b836a760c4f137ee5b0272ac8a16b345f1cac5"} Sep 09 12:04:42.447973 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.447753 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"98045cb3-41cc-4912-95d0-8b4c30178891","Type":"ContainerStarted","Data":"d3abf558c4a180c08be8c1f36287612ef118235f52b14869cb538bd6bb381af2"} Sep 09 12:04:42.815150 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.814985 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.800317471 podStartE2EDuration="2.814963581s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.64611ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:04:42.512391961 +0000 UTC m=+138.192298147" watchObservedRunningTime="2026-09-09 12:04:42.814963581 +0000 UTC m=+138.494869768" Sep 09 12:04:42.816436 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.816414 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-ffbd7df6-8r5pj"] Sep 09 12:04:42.818863 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.818848 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:42.821748 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.821726 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sc8l9\"" Sep 09 12:04:42.821999 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.821979 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 09 12:04:42.822098 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.822002 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 09 12:04:42.822319 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.822300 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 09 12:04:42.822560 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.822544 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 09 12:04:42.822677 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.822654 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 09 12:04:42.827163 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.827144 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 09 12:04:42.839861 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.839838 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-ffbd7df6-8r5pj"] Sep 09 12:04:42.910879 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.910846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ad49d01-13b7-4e87-abf6-650375a53c27-metrics-client-ca\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:42.910879 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.910883 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ad49d01-13b7-4e87-abf6-650375a53c27-serving-certs-ca-bundle\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:42.911061 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.910916 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9g22\" (UniqueName: \"kubernetes.io/projected/9ad49d01-13b7-4e87-abf6-650375a53c27-kube-api-access-n9g22\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:42.911061 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.911036 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:42.911124 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.911089 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ad49d01-13b7-4e87-abf6-650375a53c27-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:42.911157 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.911132 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-secret-telemeter-client\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:42.911232 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.911170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-telemeter-client-tls\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:42.911323 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:42.911302 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-federate-client-tls\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.012428 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.012396 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ad49d01-13b7-4e87-abf6-650375a53c27-metrics-client-ca\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.012563 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.012435 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ad49d01-13b7-4e87-abf6-650375a53c27-serving-certs-ca-bundle\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.012563 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.012456 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9g22\" (UniqueName: \"kubernetes.io/projected/9ad49d01-13b7-4e87-abf6-650375a53c27-kube-api-access-n9g22\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.012693 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.012590 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.012693 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.012668 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ad49d01-13b7-4e87-abf6-650375a53c27-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.012808 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.012743 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-secret-telemeter-client\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.012808 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.012782 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-telemeter-client-tls\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.012926 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.012828 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-federate-client-tls\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.013250 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.013196 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ad49d01-13b7-4e87-abf6-650375a53c27-metrics-client-ca\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.013389 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.013333 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ad49d01-13b7-4e87-abf6-650375a53c27-serving-certs-ca-bundle\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.013461 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.013414 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ad49d01-13b7-4e87-abf6-650375a53c27-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.014977 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.014955 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.015194 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.015169 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-telemeter-client-tls\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.015432 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.015418 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-secret-telemeter-client\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.015591 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.015572 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9ad49d01-13b7-4e87-abf6-650375a53c27-federate-client-tls\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.026682 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.026655 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9g22\" (UniqueName: \"kubernetes.io/projected/9ad49d01-13b7-4e87-abf6-650375a53c27-kube-api-access-n9g22\") pod \"telemeter-client-ffbd7df6-8r5pj\" (UID: \"9ad49d01-13b7-4e87-abf6-650375a53c27\") " pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.127893 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.127816 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" Sep 09 12:04:43.193318 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.193246 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 12:04:43.193881 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.193645 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="prometheus" containerID="cri-o://5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7" gracePeriod=600 Sep 09 12:04:43.193881 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.193795 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49" gracePeriod=600 Sep 09 12:04:43.194388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.194144 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy-web" containerID="cri-o://138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c" gracePeriod=600 Sep 09 12:04:43.194388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.194181 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="thanos-sidecar" containerID="cri-o://d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9" gracePeriod=600 Sep 09 12:04:43.194388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.194300 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy" containerID="cri-o://447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c" gracePeriod=600 Sep 09 12:04:43.194388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.194340 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="config-reloader" containerID="cri-o://72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695" gracePeriod=600 Sep 09 12:04:43.423914 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.423889 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.426567 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.426544 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-ffbd7df6-8r5pj"] Sep 09 12:04:43.434920 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:04:43.434900 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad49d01_13b7_4e87_abf6_650375a53c27.slice/crio-b4b924f6d965f51bd63da0f656188b33743af2bec92ea37232bda702e2c9351a WatchSource:0}: Error finding container b4b924f6d965f51bd63da0f656188b33743af2bec92ea37232bda702e2c9351a: Status 404 returned error can't find the container with id b4b924f6d965f51bd63da0f656188b33743af2bec92ea37232bda702e2c9351a Sep 09 12:04:43.455091 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455070 2556 generic.go:356] "Generic (PLEG): container finished" podID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerID="2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49" exitCode=0 Sep 09 12:04:43.455091 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455089 2556 generic.go:356] "Generic (PLEG): container finished" podID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerID="447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c" exitCode=0 Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455097 2556 generic.go:356] "Generic (PLEG): container finished" podID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerID="138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c" exitCode=0 Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455103 2556 generic.go:356] "Generic (PLEG): container finished" podID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerID="d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9" exitCode=0 Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455108 2556 generic.go:356] "Generic (PLEG): container finished" podID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerID="72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695" exitCode=0 Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455113 2556 generic.go:356] "Generic (PLEG): container finished" podID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerID="5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7" exitCode=0 Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455145 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455145 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerDied","Data":"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49"} Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455277 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerDied","Data":"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c"} Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455296 2556 scope.go:117] "RemoveContainer" containerID="2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49" Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455314 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerDied","Data":"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c"} Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455340 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerDied","Data":"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9"} Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455371 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerDied","Data":"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695"} Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455399 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerDied","Data":"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7"} Sep 09 12:04:43.455487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.455424 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff","Type":"ContainerDied","Data":"10f0942d8e96d5206515a7cc884ee472e006ac5eabe77531c215fd7089d55507"} Sep 09 12:04:43.456391 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.456366 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" event={"ID":"9ad49d01-13b7-4e87-abf6-650375a53c27","Type":"ContainerStarted","Data":"b4b924f6d965f51bd63da0f656188b33743af2bec92ea37232bda702e2c9351a"} Sep 09 12:04:43.462148 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.462132 2556 scope.go:117] "RemoveContainer" containerID="447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c" Sep 09 12:04:43.468775 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.468758 2556 scope.go:117] "RemoveContainer" containerID="138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c" Sep 09 12:04:43.474930 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.474913 2556 scope.go:117] "RemoveContainer" containerID="d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9" Sep 09 12:04:43.481172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.481131 2556 scope.go:117] "RemoveContainer" containerID="72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695" Sep 09 12:04:43.488315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.488298 2556 scope.go:117] "RemoveContainer" containerID="5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7" Sep 09 12:04:43.495328 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.495301 2556 scope.go:117] "RemoveContainer" containerID="8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0" Sep 09 12:04:43.501645 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.501625 2556 scope.go:117] "RemoveContainer" containerID="2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49" Sep 09 12:04:43.501897 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:43.501879 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": container with ID starting with 2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49 not found: ID does not exist" containerID="2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49" Sep 09 12:04:43.501950 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.501908 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49"} err="failed to get container status \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": rpc error: code = NotFound desc = could not find container \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": container with ID starting with 2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49 not found: ID does not exist" Sep 09 12:04:43.501950 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.501925 2556 scope.go:117] "RemoveContainer" containerID="447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c" Sep 09 12:04:43.502156 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:43.502140 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": container with ID starting with 447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c not found: ID does not exist" containerID="447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c" Sep 09 12:04:43.502197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.502161 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c"} err="failed to get container status \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": rpc error: code = NotFound desc = could not find container \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": container with ID starting with 447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c not found: ID does not exist" Sep 09 12:04:43.502197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.502175 2556 scope.go:117] "RemoveContainer" containerID="138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c" Sep 09 12:04:43.502591 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:43.502448 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": container with ID starting with 138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c not found: ID does not exist" containerID="138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c" Sep 09 12:04:43.502591 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.502472 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c"} err="failed to get container status \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": rpc error: code = NotFound desc = could not find container \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": container with ID starting with 138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c not found: ID does not exist" Sep 09 12:04:43.502591 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.502490 2556 scope.go:117] "RemoveContainer" containerID="d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9" Sep 09 12:04:43.502816 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:43.502799 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": container with ID starting with d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9 not found: ID does not exist" containerID="d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9" Sep 09 12:04:43.502853 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.502822 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9"} err="failed to get container status \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": rpc error: code = NotFound desc = could not find container \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": container with ID starting with d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9 not found: ID does not exist" Sep 09 12:04:43.502853 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.502837 2556 scope.go:117] "RemoveContainer" containerID="72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695" Sep 09 12:04:43.503126 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:43.503098 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": container with ID starting with 72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695 not found: ID does not exist" containerID="72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695" Sep 09 12:04:43.503126 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.503129 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695"} err="failed to get container status \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": rpc error: code = NotFound desc = could not find container \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": container with ID starting with 72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695 not found: ID does not exist" Sep 09 12:04:43.503284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.503142 2556 scope.go:117] "RemoveContainer" containerID="5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7" Sep 09 12:04:43.503409 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:43.503390 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": container with ID starting with 5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7 not found: ID does not exist" containerID="5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7" Sep 09 12:04:43.503464 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.503414 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7"} err="failed to get container status \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": rpc error: code = NotFound desc = could not find container \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": container with ID starting with 5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7 not found: ID does not exist" Sep 09 12:04:43.503464 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.503429 2556 scope.go:117] "RemoveContainer" containerID="8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0" Sep 09 12:04:43.503703 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:04:43.503687 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": container with ID starting with 8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0 not found: ID does not exist" containerID="8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0" Sep 09 12:04:43.503807 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.503705 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0"} err="failed to get container status \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": rpc error: code = NotFound desc = could not find container \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": container with ID starting with 8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0 not found: ID does not exist" Sep 09 12:04:43.503807 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.503718 2556 scope.go:117] "RemoveContainer" containerID="2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49" Sep 09 12:04:43.504045 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.504022 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49"} err="failed to get container status \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": rpc error: code = NotFound desc = could not find container \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": container with ID starting with 2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49 not found: ID does not exist" Sep 09 12:04:43.504129 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.504045 2556 scope.go:117] "RemoveContainer" containerID="447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c" Sep 09 12:04:43.504379 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.504352 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c"} err="failed to get container status \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": rpc error: code = NotFound desc = could not find container \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": container with ID starting with 447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c not found: ID does not exist" Sep 09 12:04:43.504379 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.504377 2556 scope.go:117] "RemoveContainer" containerID="138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c" Sep 09 12:04:43.504687 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.504666 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c"} err="failed to get container status \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": rpc error: code = NotFound desc = could not find container \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": container with ID starting with 138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c not found: ID does not exist" Sep 09 12:04:43.504764 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.504740 2556 scope.go:117] "RemoveContainer" containerID="d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9" Sep 09 12:04:43.505048 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.505015 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9"} err="failed to get container status \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": rpc error: code = NotFound desc = could not find container \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": container with ID starting with d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9 not found: ID does not exist" Sep 09 12:04:43.505136 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.505051 2556 scope.go:117] "RemoveContainer" containerID="72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695" Sep 09 12:04:43.505361 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.505343 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695"} err="failed to get container status \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": rpc error: code = NotFound desc = could not find container \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": container with ID starting with 72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695 not found: ID does not exist" Sep 09 12:04:43.505429 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.505361 2556 scope.go:117] "RemoveContainer" containerID="5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7" Sep 09 12:04:43.505620 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.505598 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7"} err="failed to get container status \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": rpc error: code = NotFound desc = could not find container \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": container with ID starting with 5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7 not found: ID does not exist" Sep 09 12:04:43.505620 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.505615 2556 scope.go:117] "RemoveContainer" containerID="8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0" Sep 09 12:04:43.505871 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.505850 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0"} err="failed to get container status \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": rpc error: code = NotFound desc = could not find container \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": container with ID starting with 8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0 not found: ID does not exist" Sep 09 12:04:43.505871 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.505870 2556 scope.go:117] "RemoveContainer" containerID="2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49" Sep 09 12:04:43.506129 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.506106 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49"} err="failed to get container status \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": rpc error: code = NotFound desc = could not find container \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": container with ID starting with 2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49 not found: ID does not exist" Sep 09 12:04:43.506196 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.506132 2556 scope.go:117] "RemoveContainer" containerID="447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c" Sep 09 12:04:43.506614 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.506589 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c"} err="failed to get container status \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": rpc error: code = NotFound desc = could not find container \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": container with ID starting with 447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c not found: ID does not exist" Sep 09 12:04:43.506680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.506615 2556 scope.go:117] "RemoveContainer" containerID="138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c" Sep 09 12:04:43.506902 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.506880 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c"} err="failed to get container status \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": rpc error: code = NotFound desc = could not find container \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": container with ID starting with 138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c not found: ID does not exist" Sep 09 12:04:43.506962 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.506913 2556 scope.go:117] "RemoveContainer" containerID="d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9" Sep 09 12:04:43.507256 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.507235 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9"} err="failed to get container status \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": rpc error: code = NotFound desc = could not find container \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": container with ID starting with d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9 not found: ID does not exist" Sep 09 12:04:43.507256 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.507255 2556 scope.go:117] "RemoveContainer" containerID="72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695" Sep 09 12:04:43.507545 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.507517 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695"} err="failed to get container status \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": rpc error: code = NotFound desc = could not find container \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": container with ID starting with 72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695 not found: ID does not exist" Sep 09 12:04:43.507597 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.507548 2556 scope.go:117] "RemoveContainer" containerID="5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7" Sep 09 12:04:43.507889 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.507856 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7"} err="failed to get container status \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": rpc error: code = NotFound desc = could not find container \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": container with ID starting with 5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7 not found: ID does not exist" Sep 09 12:04:43.507973 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.507889 2556 scope.go:117] "RemoveContainer" containerID="8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0" Sep 09 12:04:43.508162 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.508141 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0"} err="failed to get container status \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": rpc error: code = NotFound desc = could not find container \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": container with ID starting with 8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0 not found: ID does not exist" Sep 09 12:04:43.508248 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.508165 2556 scope.go:117] "RemoveContainer" containerID="2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49" Sep 09 12:04:43.508503 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.508486 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49"} err="failed to get container status \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": rpc error: code = NotFound desc = could not find container \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": container with ID starting with 2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49 not found: ID does not exist" Sep 09 12:04:43.508573 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.508506 2556 scope.go:117] "RemoveContainer" containerID="447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c" Sep 09 12:04:43.508795 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.508769 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c"} err="failed to get container status \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": rpc error: code = NotFound desc = could not find container \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": container with ID starting with 447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c not found: ID does not exist" Sep 09 12:04:43.508891 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.508796 2556 scope.go:117] "RemoveContainer" containerID="138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c" Sep 09 12:04:43.509192 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.509167 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c"} err="failed to get container status \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": rpc error: code = NotFound desc = could not find container \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": container with ID starting with 138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c not found: ID does not exist" Sep 09 12:04:43.509192 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.509193 2556 scope.go:117] "RemoveContainer" containerID="d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9" Sep 09 12:04:43.509500 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.509482 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9"} err="failed to get container status \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": rpc error: code = NotFound desc = could not find container \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": container with ID starting with d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9 not found: ID does not exist" Sep 09 12:04:43.509574 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.509501 2556 scope.go:117] "RemoveContainer" containerID="72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695" Sep 09 12:04:43.509767 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.509749 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695"} err="failed to get container status \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": rpc error: code = NotFound desc = could not find container \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": container with ID starting with 72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695 not found: ID does not exist" Sep 09 12:04:43.509824 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.509768 2556 scope.go:117] "RemoveContainer" containerID="5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7" Sep 09 12:04:43.510018 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.509985 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7"} err="failed to get container status \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": rpc error: code = NotFound desc = could not find container \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": container with ID starting with 5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7 not found: ID does not exist" Sep 09 12:04:43.510018 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.510013 2556 scope.go:117] "RemoveContainer" containerID="8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0" Sep 09 12:04:43.510302 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.510284 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0"} err="failed to get container status \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": rpc error: code = NotFound desc = could not find container \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": container with ID starting with 8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0 not found: ID does not exist" Sep 09 12:04:43.510358 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.510304 2556 scope.go:117] "RemoveContainer" containerID="2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49" Sep 09 12:04:43.510592 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.510574 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49"} err="failed to get container status \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": rpc error: code = NotFound desc = could not find container \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": container with ID starting with 2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49 not found: ID does not exist" Sep 09 12:04:43.510668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.510594 2556 scope.go:117] "RemoveContainer" containerID="447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c" Sep 09 12:04:43.510831 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.510816 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c"} err="failed to get container status \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": rpc error: code = NotFound desc = could not find container \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": container with ID starting with 447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c not found: ID does not exist" Sep 09 12:04:43.510881 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.510831 2556 scope.go:117] "RemoveContainer" containerID="138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c" Sep 09 12:04:43.511080 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.511056 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c"} err="failed to get container status \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": rpc error: code = NotFound desc = could not find container \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": container with ID starting with 138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c not found: ID does not exist" Sep 09 12:04:43.511132 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.511083 2556 scope.go:117] "RemoveContainer" containerID="d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9" Sep 09 12:04:43.511421 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.511405 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9"} err="failed to get container status \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": rpc error: code = NotFound desc = could not find container \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": container with ID starting with d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9 not found: ID does not exist" Sep 09 12:04:43.511421 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.511421 2556 scope.go:117] "RemoveContainer" containerID="72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695" Sep 09 12:04:43.511679 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.511662 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695"} err="failed to get container status \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": rpc error: code = NotFound desc = could not find container \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": container with ID starting with 72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695 not found: ID does not exist" Sep 09 12:04:43.511679 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.511679 2556 scope.go:117] "RemoveContainer" containerID="5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7" Sep 09 12:04:43.511955 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.511919 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7"} err="failed to get container status \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": rpc error: code = NotFound desc = could not find container \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": container with ID starting with 5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7 not found: ID does not exist" Sep 09 12:04:43.512043 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.511954 2556 scope.go:117] "RemoveContainer" containerID="8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0" Sep 09 12:04:43.512197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.512179 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0"} err="failed to get container status \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": rpc error: code = NotFound desc = could not find container \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": container with ID starting with 8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0 not found: ID does not exist" Sep 09 12:04:43.512197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.512196 2556 scope.go:117] "RemoveContainer" containerID="2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49" Sep 09 12:04:43.512449 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.512434 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49"} err="failed to get container status \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": rpc error: code = NotFound desc = could not find container \"2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49\": container with ID starting with 2f350f9c01cadc616b44de5bd2d36b2adbc9450a9308da9bd351534b98af2d49 not found: ID does not exist" Sep 09 12:04:43.512449 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.512449 2556 scope.go:117] "RemoveContainer" containerID="447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c" Sep 09 12:04:43.512693 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.512678 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c"} err="failed to get container status \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": rpc error: code = NotFound desc = could not find container \"447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c\": container with ID starting with 447cdbe2179a59ea46a72c24558951fa2855dd599f71cf157c4ebd1ac74ef51c not found: ID does not exist" Sep 09 12:04:43.512693 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.512692 2556 scope.go:117] "RemoveContainer" containerID="138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c" Sep 09 12:04:43.512894 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.512879 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c"} err="failed to get container status \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": rpc error: code = NotFound desc = could not find container \"138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c\": container with ID starting with 138b871a7a5994d8e09d767d335b263d7fcd6f772a6131b514d871223f43116c not found: ID does not exist" Sep 09 12:04:43.512931 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.512893 2556 scope.go:117] "RemoveContainer" containerID="d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9" Sep 09 12:04:43.513085 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.513067 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9"} err="failed to get container status \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": rpc error: code = NotFound desc = could not find container \"d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9\": container with ID starting with d01c9c76a8af6d0435d09a906a8c3d567559facf3c3eca910385c25553cf18c9 not found: ID does not exist" Sep 09 12:04:43.513145 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.513087 2556 scope.go:117] "RemoveContainer" containerID="72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695" Sep 09 12:04:43.513328 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.513303 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695"} err="failed to get container status \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": rpc error: code = NotFound desc = could not find container \"72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695\": container with ID starting with 72d792942ea024e3a56f1ad4f89c46a07422ac70d9bd0d53f19a498e85bec695 not found: ID does not exist" Sep 09 12:04:43.513382 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.513328 2556 scope.go:117] "RemoveContainer" containerID="5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7" Sep 09 12:04:43.513554 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.513540 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7"} err="failed to get container status \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": rpc error: code = NotFound desc = could not find container \"5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7\": container with ID starting with 5cfb6055bd63b48f4ef87bcc1c5f16073049752394a54d70603b0b22a88ccfc7 not found: ID does not exist" Sep 09 12:04:43.513554 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.513553 2556 scope.go:117] "RemoveContainer" containerID="8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0" Sep 09 12:04:43.513729 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.513715 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0"} err="failed to get container status \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": rpc error: code = NotFound desc = could not find container \"8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0\": container with ID starting with 8637274198bbb29179a7321d2a5d9ae1882a1fa9ffbb72d9a9a86e8e894a33d0 not found: ID does not exist" Sep 09 12:04:43.517003 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.516987 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-tls\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.517084 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517021 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-serving-certs-ca-bundle\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.517084 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517044 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdx2z\" (UniqueName: \"kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-kube-api-access-xdx2z\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.517084 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517067 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-0\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.517250 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517100 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.517250 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517126 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-metrics-client-ca\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.517250 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517152 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-kubelet-serving-ca-bundle\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517435 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517502 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-trusted-ca-bundle\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517546 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-2\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517611 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517655 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-web-config\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517712 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517739 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-grpc-tls\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517773 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517800 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517809 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-tls-assets\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517820 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517857 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-thanos-prometheus-http-client-file\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517862 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:04:43.518036 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.517887 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-kube-rbac-proxy\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518775 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.518411 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config-out\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518775 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.518454 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-metrics-client-certs\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518775 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.518481 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-db\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518775 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.518524 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-1\") pod \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\" (UID: \"a7ccb7e8-afc4-470c-bfea-ba32b3e204ff\") " Sep 09 12:04:43.518975 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.518784 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.518975 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.518803 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-metrics-client-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.518975 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.518820 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.518975 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.518836 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.518975 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.518851 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.518975 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.518939 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:04:43.519358 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.519288 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:04:43.520054 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.519964 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:43.520245 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.520216 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-kube-api-access-xdx2z" (OuterVolumeSpecName: "kube-api-access-xdx2z") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "kube-api-access-xdx2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:04:43.527143 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.522014 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:04:43.527293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.527262 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:04:43.527293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.527276 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config-out" (OuterVolumeSpecName: "config-out") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:04:43.527293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.527260 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:43.527916 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.527800 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:43.528092 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.528063 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:43.528178 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.528144 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config" (OuterVolumeSpecName: "config") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:43.528342 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.528321 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:43.528395 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.528370 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:43.528467 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.528447 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:43.537854 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.537833 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-web-config" (OuterVolumeSpecName: "web-config") pod "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" (UID: "a7ccb7e8-afc4-470c-bfea-ba32b3e204ff"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:04:43.619665 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619551 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.619819 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619721 2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-web-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.619819 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619755 2556 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.619819 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619769 2556 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-grpc-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.619819 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619785 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.619819 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619801 2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-tls-assets\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.619819 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619816 2556 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.620047 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619846 2556 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-kube-rbac-proxy\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.620047 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619861 2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-config-out\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.620047 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619875 2556 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-metrics-client-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.620047 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619888 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-db\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.620047 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619897 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.620047 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619907 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.620047 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619915 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdx2z\" (UniqueName: \"kubernetes.io/projected/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-kube-api-access-xdx2z\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.620047 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.619923 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:04:43.781918 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.781892 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 12:04:43.786850 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.786830 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 12:04:43.831328 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831299 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 12:04:43.831609 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831596 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="config-reloader" Sep 09 12:04:43.831649 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831611 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="config-reloader" Sep 09 12:04:43.831649 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831623 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy" Sep 09 12:04:43.831649 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831631 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy" Sep 09 12:04:43.831649 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831638 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy-thanos" Sep 09 12:04:43.831649 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831643 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy-thanos" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831654 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="init-config-reloader" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831659 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="init-config-reloader" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831665 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="prometheus" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831670 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="prometheus" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831677 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="thanos-sidecar" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831681 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="thanos-sidecar" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831688 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy-web" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831693 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy-web" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831734 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy-web" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831740 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="config-reloader" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831748 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831756 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="kube-rbac-proxy-thanos" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831761 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="prometheus" Sep 09 12:04:43.831788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.831766 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" containerName="thanos-sidecar" Sep 09 12:04:43.836259 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.836242 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.841258 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.841239 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 12:04:43.841386 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.841239 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 12:04:43.841574 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.841554 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 12:04:43.841721 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.841706 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 12:04:43.841800 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.841791 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 12:04:43.841883 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.841871 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 12:04:43.843001 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.842983 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 12:04:43.843092 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.842988 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2prtd\"" Sep 09 12:04:43.843338 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.843321 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 12:04:43.843404 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.843343 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 12:04:43.843560 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.843539 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 12:04:43.843795 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.843782 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f99k4nrn23j15\"" Sep 09 12:04:43.847015 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.846996 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 12:04:43.847216 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.847192 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 12:04:43.849680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.849662 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 12:04:43.853806 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.853791 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 12:04:43.921699 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921668 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.921699 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921702 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c8402e2-f17b-4fb9-9903-3cef7e063b65-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.921867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921727 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.921867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921761 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.921867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921801 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.921867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921831 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.921867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921850 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.921867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921867 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921923 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921945 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921958 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921979 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm57h\" (UniqueName: \"kubernetes.io/projected/7c8402e2-f17b-4fb9-9903-3cef7e063b65-kube-api-access-cm57h\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.921999 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.922017 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-config\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.922052 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.922084 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c8402e2-f17b-4fb9-9903-3cef7e063b65-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922119 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.922101 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922452 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.922126 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:43.922452 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:43.922144 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023089 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023045 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023098 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023154 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023178 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023430 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023235 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023430 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023271 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023430 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023430 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023318 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm57h\" (UniqueName: \"kubernetes.io/projected/7c8402e2-f17b-4fb9-9903-3cef7e063b65-kube-api-access-cm57h\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023430 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023348 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023430 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023378 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-config\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023430 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023416 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023748 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023445 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c8402e2-f17b-4fb9-9903-3cef7e063b65-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023748 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023472 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023748 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023498 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023748 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023523 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023748 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023608 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023748 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023638 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c8402e2-f17b-4fb9-9903-3cef7e063b65-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023748 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023671 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.023748 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023695 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.024114 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.023930 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.024412 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.024390 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.024831 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.024806 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.027034 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.026878 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.027034 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.026941 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c8402e2-f17b-4fb9-9903-3cef7e063b65-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.027266 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.027243 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.027409 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.027387 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.028092 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.028068 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 12:04:44.028625 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.028599 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.028762 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.028741 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.029188 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.029005 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.029757 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.029716 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c8402e2-f17b-4fb9-9903-3cef7e063b65-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.031087 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.030587 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.031087 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.031040 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.032123 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.032079 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-config\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.032232 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.032185 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c8402e2-f17b-4fb9-9903-3cef7e063b65-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.032814 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.032772 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.032946 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.032912 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.033172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.033150 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.033716 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.033695 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c8402e2-f17b-4fb9-9903-3cef7e063b65-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.069999 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.069965 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm57h\" (UniqueName: \"kubernetes.io/projected/7c8402e2-f17b-4fb9-9903-3cef7e063b65-kube-api-access-cm57h\") pod \"prometheus-k8s-0\" (UID: \"7c8402e2-f17b-4fb9-9903-3cef7e063b65\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.145168 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.145134 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:44.476983 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.476951 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 12:04:44.480436 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:04:44.480408 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8402e2_f17b_4fb9_9903_3cef7e063b65.slice/crio-7a4ab3e5356d600b640e794ca5a2615363cf1caaa8e6be3e8f4d2baab90e8298 WatchSource:0}: Error finding container 7a4ab3e5356d600b640e794ca5a2615363cf1caaa8e6be3e8f4d2baab90e8298: Status 404 returned error can't find the container with id 7a4ab3e5356d600b640e794ca5a2615363cf1caaa8e6be3e8f4d2baab90e8298 Sep 09 12:04:44.816447 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:44.816371 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ccb7e8-afc4-470c-bfea-ba32b3e204ff" path="/var/lib/kubelet/pods/a7ccb7e8-afc4-470c-bfea-ba32b3e204ff/volumes" Sep 09 12:04:45.469823 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.469702 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" event={"ID":"9ad49d01-13b7-4e87-abf6-650375a53c27","Type":"ContainerStarted","Data":"1a9b1afc4f50d67a4eb96016061f91a24fe6577610972d3101c79d097eacbb69"} Sep 09 12:04:45.469823 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.469808 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" event={"ID":"9ad49d01-13b7-4e87-abf6-650375a53c27","Type":"ContainerStarted","Data":"ff9a608c3d4a5d47138a44ecb810caf0f89a473bd7ea7e8301400661794e378d"} Sep 09 12:04:45.470015 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.469841 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" event={"ID":"9ad49d01-13b7-4e87-abf6-650375a53c27","Type":"ContainerStarted","Data":"7af446cdd2d30ec3e943d14e93769d0257fe7d8211c08bcedaf3d172603cfd6f"} Sep 09 12:04:45.471369 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.471342 2556 generic.go:356] "Generic (PLEG): container finished" podID="7c8402e2-f17b-4fb9-9903-3cef7e063b65" containerID="5e9314475001db448b24f856df70791781fcfa70ce11c62e39942db04203ee1f" exitCode=0 Sep 09 12:04:45.471471 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.471420 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c8402e2-f17b-4fb9-9903-3cef7e063b65","Type":"ContainerDied","Data":"5e9314475001db448b24f856df70791781fcfa70ce11c62e39942db04203ee1f"} Sep 09 12:04:45.471509 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.471476 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c8402e2-f17b-4fb9-9903-3cef7e063b65","Type":"ContainerStarted","Data":"7a4ab3e5356d600b640e794ca5a2615363cf1caaa8e6be3e8f4d2baab90e8298"} Sep 09 12:04:45.507642 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.507599 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-ffbd7df6-8r5pj" podStartSLOduration=1.7321929759999999 podStartE2EDuration="3.50758442s" totalImagesPullingTime="1.775391444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:04:45.506252511 +0000 UTC m=+141.186158715" watchObservedRunningTime="2026-09-09 12:04:45.50758442 +0000 UTC m=+141.187490613" Sep 09 12:04:45.756589 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.756558 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f4df6c666-qxlzs"] Sep 09 12:04:45.759112 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.759093 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.763187 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.763158 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 12:04:45.763302 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.763263 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 12:04:45.763411 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.763395 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 12:04:45.763569 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.763553 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mp6gb\"" Sep 09 12:04:45.763695 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.763679 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 12:04:45.764601 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.764584 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 12:04:45.764828 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.764806 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 12:04:45.769895 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.769825 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 12:04:45.770161 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.770144 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 12:04:45.841375 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.841348 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mg8\" (UniqueName: \"kubernetes.io/projected/d49c2636-6534-48cf-acf8-c913b25a3cb9-kube-api-access-d9mg8\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.841492 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.841395 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-serving-cert\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.841492 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.841421 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-trusted-ca-bundle\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.841492 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.841483 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-config\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.841652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.841518 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-oauth-serving-cert\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.841652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.841551 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-oauth-config\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.841652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.841565 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-service-ca\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.906383 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.906357 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f4df6c666-qxlzs"] Sep 09 12:04:45.942355 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.942325 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-oauth-config\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.942355 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.942355 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-service-ca\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.942570 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.942447 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mg8\" (UniqueName: \"kubernetes.io/projected/d49c2636-6534-48cf-acf8-c913b25a3cb9-kube-api-access-d9mg8\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.942570 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.942481 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-serving-cert\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.942570 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.942508 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-trusted-ca-bundle\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.942570 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.942554 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-config\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.942774 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.942611 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-oauth-serving-cert\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.943189 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.943127 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-service-ca\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.943358 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.943322 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-config\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.943414 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.943394 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-oauth-serving-cert\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.943446 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.943431 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-trusted-ca-bundle\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.944890 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.944874 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-oauth-config\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.945070 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.945055 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-serving-cert\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:45.960040 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:45.960018 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mg8\" (UniqueName: \"kubernetes.io/projected/d49c2636-6534-48cf-acf8-c913b25a3cb9-kube-api-access-d9mg8\") pod \"console-7f4df6c666-qxlzs\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:46.080109 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.080023 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:46.365883 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.365849 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f4df6c666-qxlzs"] Sep 09 12:04:46.370305 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:04:46.370276 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49c2636_6534_48cf_acf8_c913b25a3cb9.slice/crio-6b507870380f4d982723970ba4057e50b5d3684a70b757eadaec4d133e4c9c79 WatchSource:0}: Error finding container 6b507870380f4d982723970ba4057e50b5d3684a70b757eadaec4d133e4c9c79: Status 404 returned error can't find the container with id 6b507870380f4d982723970ba4057e50b5d3684a70b757eadaec4d133e4c9c79 Sep 09 12:04:46.478255 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.478197 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4df6c666-qxlzs" event={"ID":"d49c2636-6534-48cf-acf8-c913b25a3cb9","Type":"ContainerStarted","Data":"1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2"} Sep 09 12:04:46.478406 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.478279 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4df6c666-qxlzs" event={"ID":"d49c2636-6534-48cf-acf8-c913b25a3cb9","Type":"ContainerStarted","Data":"6b507870380f4d982723970ba4057e50b5d3684a70b757eadaec4d133e4c9c79"} Sep 09 12:04:46.481837 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.481808 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c8402e2-f17b-4fb9-9903-3cef7e063b65","Type":"ContainerStarted","Data":"cb50cfbe75221634fe81bb2ed5241034241caae4d458798b31575a88aa141b07"} Sep 09 12:04:46.481956 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.481870 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c8402e2-f17b-4fb9-9903-3cef7e063b65","Type":"ContainerStarted","Data":"7cab5082a06d8d46813b387c56e7560807715f39b3e2c98334331c979623d51e"} Sep 09 12:04:46.481956 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.481900 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c8402e2-f17b-4fb9-9903-3cef7e063b65","Type":"ContainerStarted","Data":"fe59f2c80da35e1e1dc55ba7fc09b5877682fc8ead89b319b904a1309575f755"} Sep 09 12:04:46.481956 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.481935 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c8402e2-f17b-4fb9-9903-3cef7e063b65","Type":"ContainerStarted","Data":"eef1e696b2188429d1226786166505b6620277e139f7895fbd151cb2b8f24dd1"} Sep 09 12:04:46.482102 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.481969 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c8402e2-f17b-4fb9-9903-3cef7e063b65","Type":"ContainerStarted","Data":"1c372c553405183203a37b327af00f8ab368b971aee7d3e61d63cd1f37caaa40"} Sep 09 12:04:46.482102 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.482012 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c8402e2-f17b-4fb9-9903-3cef7e063b65","Type":"ContainerStarted","Data":"b73a919bc82aeeb93fcc2f8d1384608461333f31b5e7fedf3879c78f777aa073"} Sep 09 12:04:46.517365 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.517310 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f4df6c666-qxlzs" podStartSLOduration=1.517291682 podStartE2EDuration="1.517291682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:04:46.51650414 +0000 UTC m=+142.196410326" watchObservedRunningTime="2026-09-09 12:04:46.517291682 +0000 UTC m=+142.197197869" Sep 09 12:04:46.580919 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:46.580869 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.56247888 podStartE2EDuration="3.58085144s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.37256ms" isStatefulPod=false podCreationTimestamp="2026-09-09 12:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:04:46.576495662 +0000 UTC m=+142.256401848" watchObservedRunningTime="2026-09-09 12:04:46.58085144 +0000 UTC m=+142.260757627" Sep 09 12:04:49.145900 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:49.145851 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:04:56.080535 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:56.080489 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:56.081007 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:56.080559 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:56.085293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:56.085270 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:04:56.525952 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:04:56.525925 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:05:44.146290 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:05:44.146250 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:05:44.161430 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:05:44.161406 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:05:44.728655 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:05:44.728629 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 12:06:08.855851 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:08.855820 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f4df6c666-qxlzs"] Sep 09 12:06:33.874771 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:33.874708 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f4df6c666-qxlzs" podUID="d49c2636-6534-48cf-acf8-c913b25a3cb9" containerName="console" containerID="cri-o://1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2" gracePeriod=15 Sep 09 12:06:34.111120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.111094 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:06:34.221054 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.221031 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-serving-cert\") pod \"d49c2636-6534-48cf-acf8-c913b25a3cb9\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " Sep 09 12:06:34.221240 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.221064 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9mg8\" (UniqueName: \"kubernetes.io/projected/d49c2636-6534-48cf-acf8-c913b25a3cb9-kube-api-access-d9mg8\") pod \"d49c2636-6534-48cf-acf8-c913b25a3cb9\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " Sep 09 12:06:34.221240 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.221104 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-oauth-serving-cert\") pod \"d49c2636-6534-48cf-acf8-c913b25a3cb9\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " Sep 09 12:06:34.221240 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.221133 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-oauth-config\") pod \"d49c2636-6534-48cf-acf8-c913b25a3cb9\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " Sep 09 12:06:34.221240 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.221167 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-config\") pod \"d49c2636-6534-48cf-acf8-c913b25a3cb9\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " Sep 09 12:06:34.221240 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.221240 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-service-ca\") pod \"d49c2636-6534-48cf-acf8-c913b25a3cb9\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " Sep 09 12:06:34.221511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.221274 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-trusted-ca-bundle\") pod \"d49c2636-6534-48cf-acf8-c913b25a3cb9\" (UID: \"d49c2636-6534-48cf-acf8-c913b25a3cb9\") " Sep 09 12:06:34.221611 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.221582 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-config" (OuterVolumeSpecName: "console-config") pod "d49c2636-6534-48cf-acf8-c913b25a3cb9" (UID: "d49c2636-6534-48cf-acf8-c913b25a3cb9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:06:34.221713 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.221591 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d49c2636-6534-48cf-acf8-c913b25a3cb9" (UID: "d49c2636-6534-48cf-acf8-c913b25a3cb9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:06:34.221889 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.221857 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d49c2636-6534-48cf-acf8-c913b25a3cb9" (UID: "d49c2636-6534-48cf-acf8-c913b25a3cb9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:06:34.222041 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.222010 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-service-ca" (OuterVolumeSpecName: "service-ca") pod "d49c2636-6534-48cf-acf8-c913b25a3cb9" (UID: "d49c2636-6534-48cf-acf8-c913b25a3cb9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:06:34.223121 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.223100 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d49c2636-6534-48cf-acf8-c913b25a3cb9" (UID: "d49c2636-6534-48cf-acf8-c913b25a3cb9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:06:34.223217 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.223174 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d49c2636-6534-48cf-acf8-c913b25a3cb9" (UID: "d49c2636-6534-48cf-acf8-c913b25a3cb9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:06:34.223266 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.223220 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49c2636-6534-48cf-acf8-c913b25a3cb9-kube-api-access-d9mg8" (OuterVolumeSpecName: "kube-api-access-d9mg8") pod "d49c2636-6534-48cf-acf8-c913b25a3cb9" (UID: "d49c2636-6534-48cf-acf8-c913b25a3cb9"). InnerVolumeSpecName "kube-api-access-d9mg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:06:34.322345 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.322318 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-trusted-ca-bundle\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:06:34.322345 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.322340 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:06:34.322345 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.322350 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9mg8\" (UniqueName: \"kubernetes.io/projected/d49c2636-6534-48cf-acf8-c913b25a3cb9-kube-api-access-d9mg8\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:06:34.322528 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.322360 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-oauth-serving-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:06:34.322528 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.322374 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-oauth-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:06:34.322528 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.322386 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-console-config\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:06:34.322528 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.322395 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d49c2636-6534-48cf-acf8-c913b25a3cb9-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:06:34.898439 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.898407 2556 generic.go:356] "Generic (PLEG): container finished" podID="d49c2636-6534-48cf-acf8-c913b25a3cb9" containerID="1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2" exitCode=0 Sep 09 12:06:34.898871 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.898446 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4df6c666-qxlzs" event={"ID":"d49c2636-6534-48cf-acf8-c913b25a3cb9","Type":"ContainerDied","Data":"1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2"} Sep 09 12:06:34.898871 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.898462 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4df6c666-qxlzs" Sep 09 12:06:34.898871 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.898506 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4df6c666-qxlzs" event={"ID":"d49c2636-6534-48cf-acf8-c913b25a3cb9","Type":"ContainerDied","Data":"6b507870380f4d982723970ba4057e50b5d3684a70b757eadaec4d133e4c9c79"} Sep 09 12:06:34.898871 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.898536 2556 scope.go:117] "RemoveContainer" containerID="1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2" Sep 09 12:06:34.906342 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.906327 2556 scope.go:117] "RemoveContainer" containerID="1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2" Sep 09 12:06:34.906613 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:06:34.906594 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2\": container with ID starting with 1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2 not found: ID does not exist" containerID="1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2" Sep 09 12:06:34.906674 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.906620 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2"} err="failed to get container status \"1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2\": rpc error: code = NotFound desc = could not find container \"1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2\": container with ID starting with 1d1d4f3c312bf80435a205c50958896e77b6ba27d565c60f62cb88ae4eb13ab2 not found: ID does not exist" Sep 09 12:06:34.917679 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.917653 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f4df6c666-qxlzs"] Sep 09 12:06:34.921768 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:34.921744 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f4df6c666-qxlzs"] Sep 09 12:06:36.816016 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:06:36.815982 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49c2636-6534-48cf-acf8-c913b25a3cb9" path="/var/lib/kubelet/pods/d49c2636-6534-48cf-acf8-c913b25a3cb9/volumes" Sep 09 12:07:24.705787 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:07:24.705761 2556 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 12:08:31.648671 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.648635 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dbbdf8bf8-b6wwq"] Sep 09 12:08:31.649062 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.648956 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49c2636-6534-48cf-acf8-c913b25a3cb9" containerName="console" Sep 09 12:08:31.649062 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.648968 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49c2636-6534-48cf-acf8-c913b25a3cb9" containerName="console" Sep 09 12:08:31.649062 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.649017 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49c2636-6534-48cf-acf8-c913b25a3cb9" containerName="console" Sep 09 12:08:31.651313 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.651296 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.672944 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.672920 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 12:08:31.674114 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.674094 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 12:08:31.674374 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.674358 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 12:08:31.674455 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.674380 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 12:08:31.674455 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.674389 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 12:08:31.674455 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.674380 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 12:08:31.674455 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.674452 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 12:08:31.675246 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.675232 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mp6gb\"" Sep 09 12:08:31.689728 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.689703 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 12:08:31.696924 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.696907 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dbbdf8bf8-b6wwq"] Sep 09 12:08:31.762089 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.762055 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-oauth-serving-cert\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.762089 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.762091 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv9hz\" (UniqueName: \"kubernetes.io/projected/9a55c776-36da-4709-9c8b-71e32b41749e-kube-api-access-qv9hz\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.762288 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.762123 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-console-config\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.762288 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.762174 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a55c776-36da-4709-9c8b-71e32b41749e-console-oauth-config\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.762288 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.762257 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-trusted-ca-bundle\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.762288 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.762278 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a55c776-36da-4709-9c8b-71e32b41749e-console-serving-cert\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.762415 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.762302 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-service-ca\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.863452 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.863421 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-oauth-serving-cert\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.863452 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.863451 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv9hz\" (UniqueName: \"kubernetes.io/projected/9a55c776-36da-4709-9c8b-71e32b41749e-kube-api-access-qv9hz\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.863651 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.863473 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-console-config\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.863651 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.863490 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a55c776-36da-4709-9c8b-71e32b41749e-console-oauth-config\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.863651 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.863531 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-trusted-ca-bundle\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.863651 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.863549 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a55c776-36da-4709-9c8b-71e32b41749e-console-serving-cert\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.863651 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.863566 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-service-ca\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.864248 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.864226 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-service-ca\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.864364 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.864303 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-oauth-serving-cert\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.864427 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.864375 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-console-config\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.864592 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.864573 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a55c776-36da-4709-9c8b-71e32b41749e-trusted-ca-bundle\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.866017 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.865999 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a55c776-36da-4709-9c8b-71e32b41749e-console-serving-cert\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.866105 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.866017 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a55c776-36da-4709-9c8b-71e32b41749e-console-oauth-config\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.873289 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.873270 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv9hz\" (UniqueName: \"kubernetes.io/projected/9a55c776-36da-4709-9c8b-71e32b41749e-kube-api-access-qv9hz\") pod \"console-dbbdf8bf8-b6wwq\" (UID: \"9a55c776-36da-4709-9c8b-71e32b41749e\") " pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:31.959971 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:31.959933 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:32.262487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:32.262406 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dbbdf8bf8-b6wwq"] Sep 09 12:08:32.265945 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:08:32.265916 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a55c776_36da_4709_9c8b_71e32b41749e.slice/crio-34fbc9f3a1b8db0a3319fbd5c8f1ffbc851b344cf9d2358870bf9770c3fe7e80 WatchSource:0}: Error finding container 34fbc9f3a1b8db0a3319fbd5c8f1ffbc851b344cf9d2358870bf9770c3fe7e80: Status 404 returned error can't find the container with id 34fbc9f3a1b8db0a3319fbd5c8f1ffbc851b344cf9d2358870bf9770c3fe7e80 Sep 09 12:08:32.348082 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:32.348053 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dbbdf8bf8-b6wwq" event={"ID":"9a55c776-36da-4709-9c8b-71e32b41749e","Type":"ContainerStarted","Data":"e4235769651e592b28aa573d865aaf750c29fd03688778a62faa036d303f882f"} Sep 09 12:08:32.348197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:32.348107 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dbbdf8bf8-b6wwq" event={"ID":"9a55c776-36da-4709-9c8b-71e32b41749e","Type":"ContainerStarted","Data":"34fbc9f3a1b8db0a3319fbd5c8f1ffbc851b344cf9d2358870bf9770c3fe7e80"} Sep 09 12:08:32.382007 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:32.381960 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dbbdf8bf8-b6wwq" podStartSLOduration=1.381942471 podStartE2EDuration="1.381942471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:08:32.380304596 +0000 UTC m=+368.060210786" watchObservedRunningTime="2026-09-09 12:08:32.381942471 +0000 UTC m=+368.061848657" Sep 09 12:08:41.960879 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:41.960845 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:41.961277 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:41.960938 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:41.965568 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:41.965550 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:08:42.392659 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:08:42.392591 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dbbdf8bf8-b6wwq" Sep 09 12:09:13.465353 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.465317 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4"] Sep 09 12:09:13.469072 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.469054 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.472621 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.472601 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 12:09:13.472807 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.472790 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 12:09:13.472859 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.472836 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 12:09:13.472911 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.472801 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 12:09:13.472967 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.472930 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-gdtzn\"" Sep 09 12:09:13.473103 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.473085 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 12:09:13.473191 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.473148 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 12:09:13.473269 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.473227 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 09 12:09:13.484152 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.484133 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4"] Sep 09 12:09:13.503089 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.503070 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-online-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.503193 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.503094 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7a65c812-5a57-438f-9b41-442392f98d4d-feast-data\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.503193 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.503113 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-offline-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.503193 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.503139 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-ui-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.503345 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.503226 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k9xn\" (UniqueName: \"kubernetes.io/projected/7a65c812-5a57-438f-9b41-442392f98d4d-kube-api-access-6k9xn\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.503345 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.503295 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-registry-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.603869 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.603826 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-ui-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.603869 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.603883 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k9xn\" (UniqueName: \"kubernetes.io/projected/7a65c812-5a57-438f-9b41-442392f98d4d-kube-api-access-6k9xn\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.604129 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.603958 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-registry-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.604129 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:09:13.603982 2556 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 12:09:13.604129 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.604001 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-online-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.604129 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.604023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7a65c812-5a57-438f-9b41-442392f98d4d-feast-data\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.604129 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:09:13.604053 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-ui-tls podName:7a65c812-5a57-438f-9b41-442392f98d4d nodeName:}" failed. No retries permitted until 2026-09-09 12:09:14.10403109 +0000 UTC m=+409.783937319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-ui-tls") pod "feast-simple-feast-setup-54c5589f5c-4frg4" (UID: "7a65c812-5a57-438f-9b41-442392f98d4d") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 12:09:13.604129 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.604092 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-offline-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.604482 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:09:13.604180 2556 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 12:09:13.604482 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:09:13.604242 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-offline-tls podName:7a65c812-5a57-438f-9b41-442392f98d4d nodeName:}" failed. No retries permitted until 2026-09-09 12:09:14.104228916 +0000 UTC m=+409.784135082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-offline-tls") pod "feast-simple-feast-setup-54c5589f5c-4frg4" (UID: "7a65c812-5a57-438f-9b41-442392f98d4d") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 12:09:13.604482 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.604452 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7a65c812-5a57-438f-9b41-442392f98d4d-feast-data\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.606404 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.606383 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-online-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.606502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.606431 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-registry-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:13.613239 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:13.613197 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k9xn\" (UniqueName: \"kubernetes.io/projected/7a65c812-5a57-438f-9b41-442392f98d4d-kube-api-access-6k9xn\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:14.109517 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:14.109477 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-offline-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:14.109721 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:14.109532 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-ui-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:14.111870 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:14.111847 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-offline-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:14.111983 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:14.111873 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-ui-tls\") pod \"feast-simple-feast-setup-54c5589f5c-4frg4\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:14.393110 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:14.393016 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:14.698109 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:14.698085 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4"] Sep 09 12:09:14.700041 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:09:14.700014 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a65c812_5a57_438f_9b41_442392f98d4d.slice/crio-711b1d6527388fbdddc2ab444d9c1f0c85aee3781184821efa17deb5028395b6 WatchSource:0}: Error finding container 711b1d6527388fbdddc2ab444d9c1f0c85aee3781184821efa17deb5028395b6: Status 404 returned error can't find the container with id 711b1d6527388fbdddc2ab444d9c1f0c85aee3781184821efa17deb5028395b6 Sep 09 12:09:14.701938 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:14.701919 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 12:09:15.520260 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:15.520229 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerStarted","Data":"711b1d6527388fbdddc2ab444d9c1f0c85aee3781184821efa17deb5028395b6"} Sep 09 12:09:40.641681 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:40.641633 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerStarted","Data":"14e11085c3e025a8f116f584479a871a095ad270766762105d6839821c35ef3a"} Sep 09 12:09:46.674349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:46.674293 2556 generic.go:356] "Generic (PLEG): container finished" podID="7a65c812-5a57-438f-9b41-442392f98d4d" containerID="14e11085c3e025a8f116f584479a871a095ad270766762105d6839821c35ef3a" exitCode=0 Sep 09 12:09:46.674778 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:46.674364 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerDied","Data":"14e11085c3e025a8f116f584479a871a095ad270766762105d6839821c35ef3a"} Sep 09 12:09:47.680322 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:47.680280 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerStarted","Data":"8cb814e0c5efaecafd7faa0f03a845096f83cdfa06fbe4c498de11520972a9d9"} Sep 09 12:09:52.704616 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:52.704533 2556 generic.go:356] "Generic (PLEG): container finished" podID="7a65c812-5a57-438f-9b41-442392f98d4d" containerID="8cb814e0c5efaecafd7faa0f03a845096f83cdfa06fbe4c498de11520972a9d9" exitCode=0 Sep 09 12:09:52.704966 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:52.704606 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerDied","Data":"8cb814e0c5efaecafd7faa0f03a845096f83cdfa06fbe4c498de11520972a9d9"} Sep 09 12:09:52.714664 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:52.714646 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 09 12:09:53.715193 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:53.715147 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerStarted","Data":"1dc5152c6f9a6ecebf876d4cfb1f0e7bb32a2fee8b4d5a329b6fb70a70977fe7"} Sep 09 12:09:53.715671 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:53.715230 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerStarted","Data":"95fd9ac6a184650ce6707921fae54117d9d7574fa26452612a17c5467b14136a"} Sep 09 12:09:53.715671 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:53.715256 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerStarted","Data":"1751ffb0582096c6fe2742acb485186e101d5eb7ecc027207c9bf6b6e03d2e0a"} Sep 09 12:09:53.715671 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:53.715275 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerStarted","Data":"b815f73a9875dc4820373a3ca2c85d2f192c7b948c7e38fa60773cb7599f1b57"} Sep 09 12:09:53.745720 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:53.745659 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podStartSLOduration=15.001762787 podStartE2EDuration="40.745636947s" totalImagesPullingTime="25.74387416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:09:53.743620586 +0000 UTC m=+449.423526774" watchObservedRunningTime="2026-09-09 12:09:53.745636947 +0000 UTC m=+449.425543134" Sep 09 12:09:54.394009 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:54.393967 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:54.394243 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:54.394046 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:54.394243 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:54.394075 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:54.394243 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:54.394092 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:56.394249 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:56.394101 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:56.394249 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:56.394165 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:56.394249 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:56.394185 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:56.394249 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:56.394226 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:09:56.396538 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:56.396490 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Sep 09 12:09:56.396784 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:56.396520 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.22:8816: connect: connection refused" Sep 09 12:09:56.396880 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:56.396524 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Sep 09 12:09:56.396880 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:56.396562 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.22:6571: connect: connection refused" Sep 09 12:09:59.393793 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:59.393749 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Sep 09 12:09:59.394405 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:59.393750 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.22:6571: connect: connection refused" Sep 09 12:09:59.394611 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:59.393809 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Sep 09 12:09:59.394611 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:09:59.393820 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.22:8816: connect: connection refused" Sep 09 12:10:02.394844 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:10:02.394807 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:10:02.395268 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:10:02.395085 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:10:02.395614 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:10:02.395587 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:10:02.395711 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:10:02.395657 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:10:02.395711 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:10:02.395693 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:10:02.400163 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:10:02.400144 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:10:02.775596 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:10:02.775563 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:10:02.778181 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:10:02.778145 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:11:05.690703 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:05.690666 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4"] Sep 09 12:11:05.691226 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:05.691009 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="offline" containerID="cri-o://95fd9ac6a184650ce6707921fae54117d9d7574fa26452612a17c5467b14136a" gracePeriod=30 Sep 09 12:11:05.691226 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:05.691009 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="ui" containerID="cri-o://1dc5152c6f9a6ecebf876d4cfb1f0e7bb32a2fee8b4d5a329b6fb70a70977fe7" gracePeriod=30 Sep 09 12:11:05.691226 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:05.691024 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="online" containerID="cri-o://1751ffb0582096c6fe2742acb485186e101d5eb7ecc027207c9bf6b6e03d2e0a" gracePeriod=30 Sep 09 12:11:05.691226 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:05.690965 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="registry" containerID="cri-o://b815f73a9875dc4820373a3ca2c85d2f192c7b948c7e38fa60773cb7599f1b57" gracePeriod=30 Sep 09 12:11:06.117401 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.117312 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh"] Sep 09 12:11:06.121089 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.121063 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.123694 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.123669 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-4qbxq\"" Sep 09 12:11:06.139027 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.139004 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh"] Sep 09 12:11:06.198238 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.198192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-feast-data\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.198407 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.198311 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-offline-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.198407 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.198366 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-registry-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.198407 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.198399 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mms72\" (UniqueName: \"kubernetes.io/projected/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-kube-api-access-mms72\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.198515 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.198441 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-ui-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.198515 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.198470 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-online-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.299853 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.299822 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-offline-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.299985 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.299862 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-registry-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.299985 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.299883 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mms72\" (UniqueName: \"kubernetes.io/projected/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-kube-api-access-mms72\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.299985 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.299918 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-ui-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.299985 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.299946 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-online-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.300191 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.299999 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-feast-data\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.300191 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:11:06.300005 2556 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 09 12:11:06.300191 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:11:06.300099 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-registry-tls podName:e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce nodeName:}" failed. No retries permitted until 2026-09-09 12:11:06.800077214 +0000 UTC m=+522.479983400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-registry-tls") pod "feast-simple-feast-setup-54c5589f5c-8ltvh" (UID: "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce") : secret "feast-simple-feast-setup-registry-tls" not found Sep 09 12:11:06.300742 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.300716 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-feast-data\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.302527 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.302493 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-ui-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.302983 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.302962 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-offline-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.303460 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.303438 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-online-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.309621 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.309597 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mms72\" (UniqueName: \"kubernetes.io/projected/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-kube-api-access-mms72\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.805622 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.805518 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-registry-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:06.807823 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:06.807798 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-registry-tls\") pod \"feast-simple-feast-setup-54c5589f5c-8ltvh\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:07.031268 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:07.031234 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:07.032808 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:07.032784 2556 generic.go:356] "Generic (PLEG): container finished" podID="7a65c812-5a57-438f-9b41-442392f98d4d" containerID="1dc5152c6f9a6ecebf876d4cfb1f0e7bb32a2fee8b4d5a329b6fb70a70977fe7" exitCode=0 Sep 09 12:11:07.032808 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:07.032805 2556 generic.go:356] "Generic (PLEG): container finished" podID="7a65c812-5a57-438f-9b41-442392f98d4d" containerID="1751ffb0582096c6fe2742acb485186e101d5eb7ecc027207c9bf6b6e03d2e0a" exitCode=0 Sep 09 12:11:07.032955 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:07.032831 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerDied","Data":"1dc5152c6f9a6ecebf876d4cfb1f0e7bb32a2fee8b4d5a329b6fb70a70977fe7"} Sep 09 12:11:07.032955 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:07.032861 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerDied","Data":"1751ffb0582096c6fe2742acb485186e101d5eb7ecc027207c9bf6b6e03d2e0a"} Sep 09 12:11:07.308834 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:07.308804 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh"] Sep 09 12:11:07.309305 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:11:07.309282 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5a632b2_1f79_4f28_b2f1_7760fa6bf8ce.slice/crio-df8a478914925a9c624e7560214617831557fb62911310c537f59b9e948f8a06 WatchSource:0}: Error finding container df8a478914925a9c624e7560214617831557fb62911310c537f59b9e948f8a06: Status 404 returned error can't find the container with id df8a478914925a9c624e7560214617831557fb62911310c537f59b9e948f8a06 Sep 09 12:11:08.039013 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:08.038981 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerStarted","Data":"4d00f6fc6ef1bbfe09851b7d07c00240a5b868183819b9cc92bfef0949965ac7"} Sep 09 12:11:08.039388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:08.039021 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerStarted","Data":"df8a478914925a9c624e7560214617831557fb62911310c537f59b9e948f8a06"} Sep 09 12:11:12.775191 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:12.775145 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Sep 09 12:11:12.775191 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:12.775182 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Sep 09 12:11:13.061913 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:13.061826 2556 generic.go:356] "Generic (PLEG): container finished" podID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerID="4d00f6fc6ef1bbfe09851b7d07c00240a5b868183819b9cc92bfef0949965ac7" exitCode=0 Sep 09 12:11:13.061913 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:13.061897 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerDied","Data":"4d00f6fc6ef1bbfe09851b7d07c00240a5b868183819b9cc92bfef0949965ac7"} Sep 09 12:11:14.074696 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:14.074660 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerStarted","Data":"48c3a7d847e45a71b5e1a75cbe275355d9155064f4047d15bfa2179aee12b63f"} Sep 09 12:11:19.096535 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:19.096504 2556 generic.go:356] "Generic (PLEG): container finished" podID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerID="48c3a7d847e45a71b5e1a75cbe275355d9155064f4047d15bfa2179aee12b63f" exitCode=0 Sep 09 12:11:19.096908 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:19.096579 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerDied","Data":"48c3a7d847e45a71b5e1a75cbe275355d9155064f4047d15bfa2179aee12b63f"} Sep 09 12:11:20.115905 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:20.115857 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerStarted","Data":"e12a938ca195de36db73a338e77cc1558dc211a55b090677a47ccad6956d3598"} Sep 09 12:11:20.116476 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:20.115918 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerStarted","Data":"ce0e2cc0e5491ce47398d9004fb4259168ee375eaca68b41129fef8a29e2b736"} Sep 09 12:11:20.116476 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:20.115939 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerStarted","Data":"f1b8d327718816d4175540c5dc9feab0241cf60cb9bd60a474d6434129e23e32"} Sep 09 12:11:20.116476 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:20.115958 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerStarted","Data":"b00ae9902071d58dca87abbe80d5b21db22d94ad5825c7605cdaca2d1e7c110d"} Sep 09 12:11:20.146427 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:20.146365 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podStartSLOduration=14.146342575 podStartE2EDuration="14.146342575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:11:20.145058484 +0000 UTC m=+535.824964683" watchObservedRunningTime="2026-09-09 12:11:20.146342575 +0000 UTC m=+535.826248765" Sep 09 12:11:22.031656 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:22.031618 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:22.032136 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:22.031688 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:22.032136 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:22.031708 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:22.032136 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:22.031725 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:22.033837 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:22.033806 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 09 12:11:22.034068 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:22.033829 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Sep 09 12:11:22.034273 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:22.033855 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 09 12:11:22.034394 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:22.033874 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Sep 09 12:11:22.775653 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:22.775605 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Sep 09 12:11:22.775928 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:22.775676 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Sep 09 12:11:25.032025 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:25.031578 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 09 12:11:25.032025 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:25.031578 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Sep 09 12:11:25.032025 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:25.031791 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Sep 09 12:11:25.033053 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:25.033018 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 09 12:11:27.032374 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:27.032327 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:27.032760 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:27.032403 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:27.032760 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:27.032424 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:27.032760 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:27.032441 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:28.032497 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:28.032469 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:28.032870 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:28.032572 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:28.032870 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:28.032811 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:28.033368 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:28.033345 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:28.033491 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:28.033418 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:28.033643 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:28.033626 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:28.037484 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:28.037466 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:28.166725 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:28.166693 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:11:32.775146 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:32.775109 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Sep 09 12:11:32.775146 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:32.775149 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Sep 09 12:11:32.775564 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:32.775250 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:11:32.775564 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:32.775305 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:11:36.197126 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.197088 2556 generic.go:356] "Generic (PLEG): container finished" podID="7a65c812-5a57-438f-9b41-442392f98d4d" containerID="95fd9ac6a184650ce6707921fae54117d9d7574fa26452612a17c5467b14136a" exitCode=137 Sep 09 12:11:36.197126 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.197113 2556 generic.go:356] "Generic (PLEG): container finished" podID="7a65c812-5a57-438f-9b41-442392f98d4d" containerID="b815f73a9875dc4820373a3ca2c85d2f192c7b948c7e38fa60773cb7599f1b57" exitCode=137 Sep 09 12:11:36.197126 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.197132 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerDied","Data":"95fd9ac6a184650ce6707921fae54117d9d7574fa26452612a17c5467b14136a"} Sep 09 12:11:36.197559 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.197164 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerDied","Data":"b815f73a9875dc4820373a3ca2c85d2f192c7b948c7e38fa60773cb7599f1b57"} Sep 09 12:11:36.332046 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.332022 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:11:36.389566 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.389525 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7a65c812-5a57-438f-9b41-442392f98d4d-feast-data\") pod \"7a65c812-5a57-438f-9b41-442392f98d4d\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " Sep 09 12:11:36.389712 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.389580 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-online-tls\") pod \"7a65c812-5a57-438f-9b41-442392f98d4d\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " Sep 09 12:11:36.389712 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.389611 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-ui-tls\") pod \"7a65c812-5a57-438f-9b41-442392f98d4d\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " Sep 09 12:11:36.389712 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.389697 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-registry-tls\") pod \"7a65c812-5a57-438f-9b41-442392f98d4d\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " Sep 09 12:11:36.389872 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.389763 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-offline-tls\") pod \"7a65c812-5a57-438f-9b41-442392f98d4d\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " Sep 09 12:11:36.389872 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.389802 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k9xn\" (UniqueName: \"kubernetes.io/projected/7a65c812-5a57-438f-9b41-442392f98d4d-kube-api-access-6k9xn\") pod \"7a65c812-5a57-438f-9b41-442392f98d4d\" (UID: \"7a65c812-5a57-438f-9b41-442392f98d4d\") " Sep 09 12:11:36.390059 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.390022 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a65c812-5a57-438f-9b41-442392f98d4d-feast-data" (OuterVolumeSpecName: "feast-data") pod "7a65c812-5a57-438f-9b41-442392f98d4d" (UID: "7a65c812-5a57-438f-9b41-442392f98d4d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:11:36.391863 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.391822 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7a65c812-5a57-438f-9b41-442392f98d4d" (UID: "7a65c812-5a57-438f-9b41-442392f98d4d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:11:36.391979 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.391867 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7a65c812-5a57-438f-9b41-442392f98d4d" (UID: "7a65c812-5a57-438f-9b41-442392f98d4d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:11:36.391979 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.391885 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-online-tls" (OuterVolumeSpecName: "online-tls") pod "7a65c812-5a57-438f-9b41-442392f98d4d" (UID: "7a65c812-5a57-438f-9b41-442392f98d4d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:11:36.392090 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.392015 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "7a65c812-5a57-438f-9b41-442392f98d4d" (UID: "7a65c812-5a57-438f-9b41-442392f98d4d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:11:36.392090 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.392076 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a65c812-5a57-438f-9b41-442392f98d4d-kube-api-access-6k9xn" (OuterVolumeSpecName: "kube-api-access-6k9xn") pod "7a65c812-5a57-438f-9b41-442392f98d4d" (UID: "7a65c812-5a57-438f-9b41-442392f98d4d"). InnerVolumeSpecName "kube-api-access-6k9xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:11:36.491366 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.491286 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-registry-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:11:36.491366 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.491314 2556 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-offline-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:11:36.491366 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.491324 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k9xn\" (UniqueName: \"kubernetes.io/projected/7a65c812-5a57-438f-9b41-442392f98d4d-kube-api-access-6k9xn\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:11:36.491366 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.491333 2556 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7a65c812-5a57-438f-9b41-442392f98d4d-feast-data\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:11:36.491366 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.491362 2556 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-online-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:11:36.491366 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:36.491370 2556 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7a65c812-5a57-438f-9b41-442392f98d4d-ui-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:11:37.203992 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:37.203963 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" Sep 09 12:11:37.203992 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:37.203975 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4" event={"ID":"7a65c812-5a57-438f-9b41-442392f98d4d","Type":"ContainerDied","Data":"711b1d6527388fbdddc2ab444d9c1f0c85aee3781184821efa17deb5028395b6"} Sep 09 12:11:37.204535 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:37.204026 2556 scope.go:117] "RemoveContainer" containerID="1dc5152c6f9a6ecebf876d4cfb1f0e7bb32a2fee8b4d5a329b6fb70a70977fe7" Sep 09 12:11:37.212151 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:37.212135 2556 scope.go:117] "RemoveContainer" containerID="95fd9ac6a184650ce6707921fae54117d9d7574fa26452612a17c5467b14136a" Sep 09 12:11:37.221098 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:37.220964 2556 scope.go:117] "RemoveContainer" containerID="1751ffb0582096c6fe2742acb485186e101d5eb7ecc027207c9bf6b6e03d2e0a" Sep 09 12:11:37.225162 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:37.225142 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4"] Sep 09 12:11:37.228830 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:37.228795 2556 scope.go:117] "RemoveContainer" containerID="b815f73a9875dc4820373a3ca2c85d2f192c7b948c7e38fa60773cb7599f1b57" Sep 09 12:11:37.230833 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:37.230815 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-4frg4"] Sep 09 12:11:37.235981 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:37.235963 2556 scope.go:117] "RemoveContainer" containerID="8cb814e0c5efaecafd7faa0f03a845096f83cdfa06fbe4c498de11520972a9d9" Sep 09 12:11:37.242989 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:37.242974 2556 scope.go:117] "RemoveContainer" containerID="14e11085c3e025a8f116f584479a871a095ad270766762105d6839821c35ef3a" Sep 09 12:11:38.816127 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:11:38.816093 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" path="/var/lib/kubelet/pods/7a65c812-5a57-438f-9b41-442392f98d4d/volumes" Sep 09 12:12:31.223038 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223003 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt"] Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223361 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="online" Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223373 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="online" Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223388 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="registry" Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223394 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="registry" Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223401 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="offline" Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223408 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="offline" Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223417 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="feast-init" Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223423 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="feast-init" Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223436 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="feast-apply" Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223441 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="feast-apply" Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223446 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="ui" Sep 09 12:12:31.223457 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223451 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="ui" Sep 09 12:12:31.223808 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223505 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="online" Sep 09 12:12:31.223808 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223515 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="ui" Sep 09 12:12:31.223808 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223521 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="registry" Sep 09 12:12:31.223808 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.223528 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a65c812-5a57-438f-9b41-442392f98d4d" containerName="offline" Sep 09 12:12:31.228050 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.228030 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.231229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.231041 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 12:12:31.231229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.231089 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 12:12:31.231229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.231113 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 12:12:31.231229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.231129 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 12:12:31.231229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.231121 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 12:12:31.231229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.231166 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 12:12:31.231573 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.231349 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-6w55f\"" Sep 09 12:12:31.231573 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.231352 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 09 12:12:31.240068 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.240040 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt"] Sep 09 12:12:31.378921 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.378889 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-offline-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.379100 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.378930 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-online-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.379100 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.379031 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-ui-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.379100 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.379076 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/aa256345-a534-40c6-ad06-176286c5ab14-registry-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.379243 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.379131 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aa256345-a534-40c6-ad06-176286c5ab14-feast-data\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.379243 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.379149 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwmhm\" (UniqueName: \"kubernetes.io/projected/aa256345-a534-40c6-ad06-176286c5ab14-kube-api-access-vwmhm\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.480516 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.480418 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-ui-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.480516 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.480482 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/aa256345-a534-40c6-ad06-176286c5ab14-registry-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.480745 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.480547 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aa256345-a534-40c6-ad06-176286c5ab14-feast-data\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.480745 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.480571 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwmhm\" (UniqueName: \"kubernetes.io/projected/aa256345-a534-40c6-ad06-176286c5ab14-kube-api-access-vwmhm\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.480745 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.480615 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-offline-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.480745 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.480656 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-online-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.481040 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.481007 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aa256345-a534-40c6-ad06-176286c5ab14-feast-data\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.481258 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.481236 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/aa256345-a534-40c6-ad06-176286c5ab14-registry-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.482952 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.482932 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-online-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.483461 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.483439 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-ui-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.483535 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.483439 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-offline-tls\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.490868 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.490846 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwmhm\" (UniqueName: \"kubernetes.io/projected/aa256345-a534-40c6-ad06-176286c5ab14-kube-api-access-vwmhm\") pod \"feast-simple-feast-remote-setup-f78788578-ws7zt\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.538737 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.538700 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:12:31.815154 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:31.815114 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt"] Sep 09 12:12:31.817156 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:12:31.817127 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa256345_a534_40c6_ad06_176286c5ab14.slice/crio-ec652e5267aedc8964e46753e7cee2b3090b380cc3ff2a8f13e4d66c16a0a1e7 WatchSource:0}: Error finding container ec652e5267aedc8964e46753e7cee2b3090b380cc3ff2a8f13e4d66c16a0a1e7: Status 404 returned error can't find the container with id ec652e5267aedc8964e46753e7cee2b3090b380cc3ff2a8f13e4d66c16a0a1e7 Sep 09 12:12:32.428897 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:32.428859 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerStarted","Data":"1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e"} Sep 09 12:12:32.429289 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:32.428913 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerStarted","Data":"ec652e5267aedc8964e46753e7cee2b3090b380cc3ff2a8f13e4d66c16a0a1e7"} Sep 09 12:12:37.452609 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:37.452577 2556 generic.go:356] "Generic (PLEG): container finished" podID="aa256345-a534-40c6-ad06-176286c5ab14" containerID="1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e" exitCode=0 Sep 09 12:12:37.452994 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:37.452629 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerDied","Data":"1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e"} Sep 09 12:12:38.458662 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:38.458626 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerStarted","Data":"03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb"} Sep 09 12:12:59.546587 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:59.546553 2556 generic.go:356] "Generic (PLEG): container finished" podID="aa256345-a534-40c6-ad06-176286c5ab14" containerID="03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb" exitCode=0 Sep 09 12:12:59.546587 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:12:59.546595 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerDied","Data":"03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb"} Sep 09 12:13:00.554926 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:00.554892 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerStarted","Data":"38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d"} Sep 09 12:13:00.555414 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:00.554943 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerStarted","Data":"da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3"} Sep 09 12:13:00.555414 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:00.554960 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerStarted","Data":"bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b"} Sep 09 12:13:00.581536 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:00.581470 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podStartSLOduration=29.581447095 podStartE2EDuration="29.581447095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:13:00.581306455 +0000 UTC m=+636.261212643" watchObservedRunningTime="2026-09-09 12:13:00.581447095 +0000 UTC m=+636.261353283" Sep 09 12:13:01.539193 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:01.539148 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:13:01.539421 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:01.539241 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:13:01.539421 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:01.539260 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:13:01.539421 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:01.539278 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:13:01.539421 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:01.539294 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:13:01.539421 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:01.539311 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:13:01.541025 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:01.540966 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 09 12:13:01.541025 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:01.541004 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused" Sep 09 12:13:01.541246 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:01.540983 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 12:13:04.539366 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:04.539317 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused" Sep 09 12:13:04.539366 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:04.539373 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 12:13:04.539918 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:04.539317 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 09 12:13:07.539969 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:07.539924 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 09 12:13:07.540548 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:07.540525 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:13:07.541340 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:07.541319 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:13:07.600689 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:07.600648 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:13:08.613037 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:08.613007 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:13:10.539986 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:10.539956 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:13:10.602595 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:13:10.602568 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:14:13.557195 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:13.557163 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt"] Sep 09 12:14:13.559829 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:13.557420 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="online" containerID="cri-o://bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b" gracePeriod=30 Sep 09 12:14:13.559829 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:13.557652 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="ui" containerID="cri-o://38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d" gracePeriod=30 Sep 09 12:14:13.559829 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:13.557704 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="offline" containerID="cri-o://da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3" gracePeriod=30 Sep 09 12:14:13.663373 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:13.663342 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh"] Sep 09 12:14:13.663741 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:13.663579 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="registry" containerID="cri-o://b00ae9902071d58dca87abbe80d5b21db22d94ad5825c7605cdaca2d1e7c110d" gracePeriod=30 Sep 09 12:14:13.663741 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:13.663642 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="online" containerID="cri-o://f1b8d327718816d4175540c5dc9feab0241cf60cb9bd60a474d6434129e23e32" gracePeriod=30 Sep 09 12:14:13.663741 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:13.663668 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="offline" containerID="cri-o://ce0e2cc0e5491ce47398d9004fb4259168ee375eaca68b41129fef8a29e2b736" gracePeriod=30 Sep 09 12:14:13.663741 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:13.663665 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="ui" containerID="cri-o://e12a938ca195de36db73a338e77cc1558dc211a55b090677a47ccad6956d3598" gracePeriod=30 Sep 09 12:14:14.863057 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:14.862977 2556 generic.go:356] "Generic (PLEG): container finished" podID="aa256345-a534-40c6-ad06-176286c5ab14" containerID="38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d" exitCode=0 Sep 09 12:14:14.863057 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:14.863000 2556 generic.go:356] "Generic (PLEG): container finished" podID="aa256345-a534-40c6-ad06-176286c5ab14" containerID="bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b" exitCode=0 Sep 09 12:14:14.863057 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:14.863052 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerDied","Data":"38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d"} Sep 09 12:14:14.863600 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:14.863095 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerDied","Data":"bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b"} Sep 09 12:14:14.866095 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:14.866075 2556 generic.go:356] "Generic (PLEG): container finished" podID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerID="e12a938ca195de36db73a338e77cc1558dc211a55b090677a47ccad6956d3598" exitCode=0 Sep 09 12:14:14.866095 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:14.866092 2556 generic.go:356] "Generic (PLEG): container finished" podID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerID="f1b8d327718816d4175540c5dc9feab0241cf60cb9bd60a474d6434129e23e32" exitCode=0 Sep 09 12:14:14.866309 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:14.866130 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerDied","Data":"e12a938ca195de36db73a338e77cc1558dc211a55b090677a47ccad6956d3598"} Sep 09 12:14:14.866309 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:14.866154 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerDied","Data":"f1b8d327718816d4175540c5dc9feab0241cf60cb9bd60a474d6434129e23e32"} Sep 09 12:14:18.034320 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:18.034227 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 09 12:14:18.164135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:18.164103 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 09 12:14:18.594177 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:18.594134 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 12:14:20.602684 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:20.602642 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 09 12:14:28.034093 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:28.034038 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 09 12:14:28.164001 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:28.163966 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 09 12:14:28.594788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:28.594746 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 12:14:30.602805 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:30.602763 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 09 12:14:38.033405 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:38.033361 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 09 12:14:38.033858 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:38.033515 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:14:38.164305 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:38.164266 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 09 12:14:38.164494 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:38.164384 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:14:38.594421 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:38.594373 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 12:14:38.594588 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:38.594514 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:14:40.602491 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:40.602450 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 09 12:14:40.602849 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:40.602560 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:14:43.786807 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.786783 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:14:43.819548 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.819482 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-online-tls\") pod \"aa256345-a534-40c6-ad06-176286c5ab14\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " Sep 09 12:14:43.819548 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.819515 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/aa256345-a534-40c6-ad06-176286c5ab14-registry-tls\") pod \"aa256345-a534-40c6-ad06-176286c5ab14\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " Sep 09 12:14:43.819740 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.819556 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-ui-tls\") pod \"aa256345-a534-40c6-ad06-176286c5ab14\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " Sep 09 12:14:43.819740 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.819642 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-offline-tls\") pod \"aa256345-a534-40c6-ad06-176286c5ab14\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " Sep 09 12:14:43.819740 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.819689 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwmhm\" (UniqueName: \"kubernetes.io/projected/aa256345-a534-40c6-ad06-176286c5ab14-kube-api-access-vwmhm\") pod \"aa256345-a534-40c6-ad06-176286c5ab14\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " Sep 09 12:14:43.819740 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.819737 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aa256345-a534-40c6-ad06-176286c5ab14-feast-data\") pod \"aa256345-a534-40c6-ad06-176286c5ab14\" (UID: \"aa256345-a534-40c6-ad06-176286c5ab14\") " Sep 09 12:14:43.819955 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.819894 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa256345-a534-40c6-ad06-176286c5ab14-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "aa256345-a534-40c6-ad06-176286c5ab14" (UID: "aa256345-a534-40c6-ad06-176286c5ab14"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:14:43.820540 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.820501 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa256345-a534-40c6-ad06-176286c5ab14-feast-data" (OuterVolumeSpecName: "feast-data") pod "aa256345-a534-40c6-ad06-176286c5ab14" (UID: "aa256345-a534-40c6-ad06-176286c5ab14"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:14:43.821872 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.821849 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-online-tls" (OuterVolumeSpecName: "online-tls") pod "aa256345-a534-40c6-ad06-176286c5ab14" (UID: "aa256345-a534-40c6-ad06-176286c5ab14"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:14:43.821961 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.821873 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "aa256345-a534-40c6-ad06-176286c5ab14" (UID: "aa256345-a534-40c6-ad06-176286c5ab14"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:14:43.822079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.822060 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "aa256345-a534-40c6-ad06-176286c5ab14" (UID: "aa256345-a534-40c6-ad06-176286c5ab14"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:14:43.822174 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.822077 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa256345-a534-40c6-ad06-176286c5ab14-kube-api-access-vwmhm" (OuterVolumeSpecName: "kube-api-access-vwmhm") pod "aa256345-a534-40c6-ad06-176286c5ab14" (UID: "aa256345-a534-40c6-ad06-176286c5ab14"). InnerVolumeSpecName "kube-api-access-vwmhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:14:43.920897 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.920867 2556 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-offline-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:43.920897 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.920892 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwmhm\" (UniqueName: \"kubernetes.io/projected/aa256345-a534-40c6-ad06-176286c5ab14-kube-api-access-vwmhm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:43.920897 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.920901 2556 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/aa256345-a534-40c6-ad06-176286c5ab14-feast-data\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:43.921111 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.920910 2556 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-online-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:43.921111 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.920920 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/aa256345-a534-40c6-ad06-176286c5ab14-registry-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:43.921111 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.920927 2556 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/aa256345-a534-40c6-ad06-176286c5ab14-ui-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:43.983639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.983600 2556 generic.go:356] "Generic (PLEG): container finished" podID="aa256345-a534-40c6-ad06-176286c5ab14" containerID="da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3" exitCode=137 Sep 09 12:14:43.983789 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.983661 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" Sep 09 12:14:43.983789 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.983690 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerDied","Data":"da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3"} Sep 09 12:14:43.983789 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.983739 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt" event={"ID":"aa256345-a534-40c6-ad06-176286c5ab14","Type":"ContainerDied","Data":"ec652e5267aedc8964e46753e7cee2b3090b380cc3ff2a8f13e4d66c16a0a1e7"} Sep 09 12:14:43.983945 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.983760 2556 scope.go:117] "RemoveContainer" containerID="38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d" Sep 09 12:14:43.987100 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.987078 2556 generic.go:356] "Generic (PLEG): container finished" podID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerID="ce0e2cc0e5491ce47398d9004fb4259168ee375eaca68b41129fef8a29e2b736" exitCode=137 Sep 09 12:14:43.987100 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.987096 2556 generic.go:356] "Generic (PLEG): container finished" podID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerID="b00ae9902071d58dca87abbe80d5b21db22d94ad5825c7605cdaca2d1e7c110d" exitCode=137 Sep 09 12:14:43.987295 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.987116 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerDied","Data":"ce0e2cc0e5491ce47398d9004fb4259168ee375eaca68b41129fef8a29e2b736"} Sep 09 12:14:43.987295 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.987165 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerDied","Data":"b00ae9902071d58dca87abbe80d5b21db22d94ad5825c7605cdaca2d1e7c110d"} Sep 09 12:14:43.992516 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:43.992503 2556 scope.go:117] "RemoveContainer" containerID="da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3" Sep 09 12:14:44.000090 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.000074 2556 scope.go:117] "RemoveContainer" containerID="bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b" Sep 09 12:14:44.005752 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.005731 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt"] Sep 09 12:14:44.007914 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.007894 2556 scope.go:117] "RemoveContainer" containerID="03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb" Sep 09 12:14:44.011382 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.011360 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-f78788578-ws7zt"] Sep 09 12:14:44.015283 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.015265 2556 scope.go:117] "RemoveContainer" containerID="1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e" Sep 09 12:14:44.024268 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.024250 2556 scope.go:117] "RemoveContainer" containerID="38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d" Sep 09 12:14:44.024790 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:14:44.024771 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d\": container with ID starting with 38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d not found: ID does not exist" containerID="38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d" Sep 09 12:14:44.024858 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.024802 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d"} err="failed to get container status \"38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d\": rpc error: code = NotFound desc = could not find container \"38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d\": container with ID starting with 38862b07b0f27b87923070e60692262eb6b0ff00e5e1ef45c5101ee61fc2022d not found: ID does not exist" Sep 09 12:14:44.024858 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.024822 2556 scope.go:117] "RemoveContainer" containerID="da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3" Sep 09 12:14:44.025085 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:14:44.025071 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3\": container with ID starting with da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3 not found: ID does not exist" containerID="da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3" Sep 09 12:14:44.025140 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.025089 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3"} err="failed to get container status \"da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3\": rpc error: code = NotFound desc = could not find container \"da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3\": container with ID starting with da7b16798cf8af55806dcd8da0d0af898f811f250914437ea6ad700f4b0b13f3 not found: ID does not exist" Sep 09 12:14:44.025140 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.025101 2556 scope.go:117] "RemoveContainer" containerID="bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b" Sep 09 12:14:44.025354 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:14:44.025339 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b\": container with ID starting with bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b not found: ID does not exist" containerID="bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b" Sep 09 12:14:44.025399 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.025357 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b"} err="failed to get container status \"bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b\": rpc error: code = NotFound desc = could not find container \"bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b\": container with ID starting with bdd92cc71928ed91c26135e9e6db027807d1c318c7bd10fd341c7a49368f074b not found: ID does not exist" Sep 09 12:14:44.025399 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.025370 2556 scope.go:117] "RemoveContainer" containerID="03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb" Sep 09 12:14:44.025568 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:14:44.025554 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb\": container with ID starting with 03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb not found: ID does not exist" containerID="03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb" Sep 09 12:14:44.025608 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.025574 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb"} err="failed to get container status \"03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb\": rpc error: code = NotFound desc = could not find container \"03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb\": container with ID starting with 03b85e955907f5fb410526423dced15378a15693f5cb22083ca33d6ae4b96fdb not found: ID does not exist" Sep 09 12:14:44.025608 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.025586 2556 scope.go:117] "RemoveContainer" containerID="1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e" Sep 09 12:14:44.025816 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:14:44.025800 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e\": container with ID starting with 1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e not found: ID does not exist" containerID="1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e" Sep 09 12:14:44.025872 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.025818 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e"} err="failed to get container status \"1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e\": rpc error: code = NotFound desc = could not find container \"1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e\": container with ID starting with 1bde7e426f1d6306ddbe35018fd008e33b8255deeac562ece9b79e1ccb289e8e not found: ID does not exist" Sep 09 12:14:44.296214 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.296182 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:14:44.324369 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.324344 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-online-tls\") pod \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " Sep 09 12:14:44.324490 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.324389 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-feast-data\") pod \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " Sep 09 12:14:44.324545 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.324521 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-registry-tls\") pod \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " Sep 09 12:14:44.324633 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.324617 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-offline-tls\") pod \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " Sep 09 12:14:44.324690 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.324652 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-ui-tls\") pod \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " Sep 09 12:14:44.324690 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.324681 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mms72\" (UniqueName: \"kubernetes.io/projected/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-kube-api-access-mms72\") pod \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\" (UID: \"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce\") " Sep 09 12:14:44.324988 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.324908 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-feast-data" (OuterVolumeSpecName: "feast-data") pod "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" (UID: "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:14:44.324988 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.324981 2556 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-feast-data\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:44.327020 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.326995 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-online-tls" (OuterVolumeSpecName: "online-tls") pod "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" (UID: "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:14:44.327340 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.327304 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-kube-api-access-mms72" (OuterVolumeSpecName: "kube-api-access-mms72") pod "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" (UID: "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce"). InnerVolumeSpecName "kube-api-access-mms72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:14:44.327340 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.327331 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" (UID: "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:14:44.327510 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.327351 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" (UID: "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:14:44.327510 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.327369 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" (UID: "e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:14:44.425957 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.425910 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mms72\" (UniqueName: \"kubernetes.io/projected/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-kube-api-access-mms72\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:44.425957 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.425951 2556 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-online-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:44.425957 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.425960 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-registry-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:44.425957 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.425968 2556 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-offline-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:44.425957 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.425975 2556 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce-ui-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:14:44.816237 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.816190 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa256345-a534-40c6-ad06-176286c5ab14" path="/var/lib/kubelet/pods/aa256345-a534-40c6-ad06-176286c5ab14/volumes" Sep 09 12:14:44.994221 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.994176 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" event={"ID":"e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce","Type":"ContainerDied","Data":"df8a478914925a9c624e7560214617831557fb62911310c537f59b9e948f8a06"} Sep 09 12:14:44.994221 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.994199 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh" Sep 09 12:14:44.994466 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:44.994238 2556 scope.go:117] "RemoveContainer" containerID="e12a938ca195de36db73a338e77cc1558dc211a55b090677a47ccad6956d3598" Sep 09 12:14:45.001958 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:45.001943 2556 scope.go:117] "RemoveContainer" containerID="ce0e2cc0e5491ce47398d9004fb4259168ee375eaca68b41129fef8a29e2b736" Sep 09 12:14:45.009084 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:45.009069 2556 scope.go:117] "RemoveContainer" containerID="f1b8d327718816d4175540c5dc9feab0241cf60cb9bd60a474d6434129e23e32" Sep 09 12:14:45.015768 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:45.015739 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh"] Sep 09 12:14:45.016976 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:45.016956 2556 scope.go:117] "RemoveContainer" containerID="b00ae9902071d58dca87abbe80d5b21db22d94ad5825c7605cdaca2d1e7c110d" Sep 09 12:14:45.019180 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:45.019156 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-54c5589f5c-8ltvh"] Sep 09 12:14:45.024286 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:45.024266 2556 scope.go:117] "RemoveContainer" containerID="48c3a7d847e45a71b5e1a75cbe275355d9155064f4047d15bfa2179aee12b63f" Sep 09 12:14:45.030760 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:45.030745 2556 scope.go:117] "RemoveContainer" containerID="4d00f6fc6ef1bbfe09851b7d07c00240a5b868183819b9cc92bfef0949965ac7" Sep 09 12:14:46.815799 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:46.815767 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" path="/var/lib/kubelet/pods/e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce/volumes" Sep 09 12:14:56.620386 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.620354 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-bk7gl"] Sep 09 12:14:56.621120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621096 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="feast-apply" Sep 09 12:14:56.621120 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621122 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="feast-apply" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621134 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="online" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621143 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="online" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621168 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="offline" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621177 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="offline" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621190 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="ui" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621214 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="ui" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621230 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="registry" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621239 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="registry" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621247 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="online" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621254 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="online" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621270 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="ui" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621277 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="ui" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621287 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="feast-apply" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621295 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="feast-apply" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621308 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="offline" Sep 09 12:14:56.621315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621316 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="offline" Sep 09 12:14:56.622135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621327 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="feast-init" Sep 09 12:14:56.622135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621335 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="feast-init" Sep 09 12:14:56.622135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621349 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="feast-init" Sep 09 12:14:56.622135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621357 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="feast-init" Sep 09 12:14:56.622135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621427 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="ui" Sep 09 12:14:56.622135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621439 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="offline" Sep 09 12:14:56.622135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621447 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="offline" Sep 09 12:14:56.622135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621458 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="online" Sep 09 12:14:56.622135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621467 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa256345-a534-40c6-ad06-176286c5ab14" containerName="online" Sep 09 12:14:56.622135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621478 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="registry" Sep 09 12:14:56.622135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.621485 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a632b2-1f79-4f28-b2f1-7760fa6bf8ce" containerName="ui" Sep 09 12:14:56.624605 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.624581 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-bk7gl" Sep 09 12:14:56.627078 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.627057 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 12:14:56.628129 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.628105 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 12:14:56.628129 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.628121 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-sffg8\"" Sep 09 12:14:56.630321 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.630302 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-bk7gl"] Sep 09 12:14:56.730561 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.730530 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9sp6\" (UniqueName: \"kubernetes.io/projected/fb85721e-f0d8-4863-baf1-470284c27d7d-kube-api-access-p9sp6\") pod \"redis-69bbd798b6-bk7gl\" (UID: \"fb85721e-f0d8-4863-baf1-470284c27d7d\") " pod="test-ns-feast/redis-69bbd798b6-bk7gl" Sep 09 12:14:56.831060 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.831029 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9sp6\" (UniqueName: \"kubernetes.io/projected/fb85721e-f0d8-4863-baf1-470284c27d7d-kube-api-access-p9sp6\") pod \"redis-69bbd798b6-bk7gl\" (UID: \"fb85721e-f0d8-4863-baf1-470284c27d7d\") " pod="test-ns-feast/redis-69bbd798b6-bk7gl" Sep 09 12:14:56.842821 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.842800 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9sp6\" (UniqueName: \"kubernetes.io/projected/fb85721e-f0d8-4863-baf1-470284c27d7d-kube-api-access-p9sp6\") pod \"redis-69bbd798b6-bk7gl\" (UID: \"fb85721e-f0d8-4863-baf1-470284c27d7d\") " pod="test-ns-feast/redis-69bbd798b6-bk7gl" Sep 09 12:14:56.943871 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:56.943848 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-bk7gl" Sep 09 12:14:57.218121 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:57.218042 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-bk7gl"] Sep 09 12:14:57.220799 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:14:57.220775 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb85721e_f0d8_4863_baf1_470284c27d7d.slice/crio-47f06fdcdda2731b475853057be076b9f0b909b93f6df52bb26fb18317864b95 WatchSource:0}: Error finding container 47f06fdcdda2731b475853057be076b9f0b909b93f6df52bb26fb18317864b95: Status 404 returned error can't find the container with id 47f06fdcdda2731b475853057be076b9f0b909b93f6df52bb26fb18317864b95 Sep 09 12:14:57.222760 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:57.222743 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 12:14:58.050658 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:14:58.050615 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-bk7gl" event={"ID":"fb85721e-f0d8-4863-baf1-470284c27d7d","Type":"ContainerStarted","Data":"47f06fdcdda2731b475853057be076b9f0b909b93f6df52bb26fb18317864b95"} Sep 09 12:15:00.066013 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:00.065974 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-bk7gl" event={"ID":"fb85721e-f0d8-4863-baf1-470284c27d7d","Type":"ContainerStarted","Data":"afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f"} Sep 09 12:15:00.082235 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:00.082170 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-bk7gl" podStartSLOduration=2.039263782 podStartE2EDuration="4.082156013s" totalImagesPullingTime="2.042892231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:15:00.08155064 +0000 UTC m=+755.761456829" watchObservedRunningTime="2026-09-09 12:15:00.082156013 +0000 UTC m=+755.762062199" Sep 09 12:15:07.439078 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.439038 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx"] Sep 09 12:15:07.442848 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.442825 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.445511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.445483 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 09 12:15:07.445643 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.445521 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 09 12:15:07.445643 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.445482 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 09 12:15:07.445643 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.445592 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 09 12:15:07.445859 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.445840 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-5jq58\"" Sep 09 12:15:07.445914 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.445891 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 09 12:15:07.453844 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.453822 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx"] Sep 09 12:15:07.519963 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.519931 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-registry-tls\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.520135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.519996 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-online-tls\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.520135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.520023 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmxf\" (UniqueName: \"kubernetes.io/projected/bd059e10-664e-427a-b172-340443c18707-kube-api-access-lgmxf\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.520135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.520048 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-offline-tls\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.520135 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.520067 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bd059e10-664e-427a-b172-340443c18707-feast-data\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.621316 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.621284 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-registry-tls\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.621510 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.621339 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-online-tls\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.621510 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.621363 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmxf\" (UniqueName: \"kubernetes.io/projected/bd059e10-664e-427a-b172-340443c18707-kube-api-access-lgmxf\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.621510 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.621395 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-offline-tls\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.621510 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.621416 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bd059e10-664e-427a-b172-340443c18707-feast-data\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.621779 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.621762 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bd059e10-664e-427a-b172-340443c18707-feast-data\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.623804 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.623776 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-offline-tls\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.623907 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.623851 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-online-tls\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.623948 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.623912 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-registry-tls\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.629125 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.629106 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmxf\" (UniqueName: \"kubernetes.io/projected/bd059e10-664e-427a-b172-340443c18707-kube-api-access-lgmxf\") pod \"feast-credit-scoring-7d974d8755-h9hpx\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:07.752180 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:07.752092 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:08.029304 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:08.029272 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx"] Sep 09 12:15:08.030935 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:15:08.030906 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd059e10_664e_427a_b172_340443c18707.slice/crio-5430e238e7dbd0b2fe047a248c60d7ff9965bd4a8d3e4e08cb47409e25d9d3a5 WatchSource:0}: Error finding container 5430e238e7dbd0b2fe047a248c60d7ff9965bd4a8d3e4e08cb47409e25d9d3a5: Status 404 returned error can't find the container with id 5430e238e7dbd0b2fe047a248c60d7ff9965bd4a8d3e4e08cb47409e25d9d3a5 Sep 09 12:15:08.097877 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:08.097839 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerStarted","Data":"3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe"} Sep 09 12:15:08.097989 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:08.097923 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerStarted","Data":"5430e238e7dbd0b2fe047a248c60d7ff9965bd4a8d3e4e08cb47409e25d9d3a5"} Sep 09 12:15:10.106741 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:10.106710 2556 generic.go:356] "Generic (PLEG): container finished" podID="bd059e10-664e-427a-b172-340443c18707" containerID="3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe" exitCode=0 Sep 09 12:15:10.107103 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:10.106782 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerDied","Data":"3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe"} Sep 09 12:15:11.112480 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:11.112447 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerStarted","Data":"b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7"} Sep 09 12:15:17.140779 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:17.140688 2556 generic.go:356] "Generic (PLEG): container finished" podID="bd059e10-664e-427a-b172-340443c18707" containerID="b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7" exitCode=0 Sep 09 12:15:17.140779 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:17.140743 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerDied","Data":"b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7"} Sep 09 12:15:18.153148 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:18.153109 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerStarted","Data":"66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635"} Sep 09 12:15:18.153531 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:18.153247 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerStarted","Data":"910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33"} Sep 09 12:15:18.153531 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:18.153279 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerStarted","Data":"82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19"} Sep 09 12:15:18.181185 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:18.181127 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podStartSLOduration=11.181109451 podStartE2EDuration="11.181109451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:15:18.18089988 +0000 UTC m=+773.860806068" watchObservedRunningTime="2026-09-09 12:15:18.181109451 +0000 UTC m=+773.861015640" Sep 09 12:15:19.753044 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:19.753008 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:19.753675 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:19.753651 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:19.753772 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:19.753695 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:19.754726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:19.754698 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Sep 09 12:15:19.754850 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:19.754740 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 09 12:15:19.754850 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:19.754736 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused" Sep 09 12:15:22.752789 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:22.752746 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Sep 09 12:15:22.752789 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:22.752760 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused" Sep 09 12:15:22.753382 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:22.752808 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 09 12:15:25.753343 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:25.753291 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Sep 09 12:15:25.754138 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:25.754114 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:25.754595 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:25.754576 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:25.754983 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:25.754967 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:25.761550 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:25.761531 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:26.199441 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:26.199413 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:26.204027 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:26.204006 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:27.752349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:27.752302 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:28.753032 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:28.752993 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Sep 09 12:15:31.753213 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:31.753184 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:15:31.754050 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:15:31.754031 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:16:38.585152 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:38.585115 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx"] Sep 09 12:16:38.585618 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:38.585421 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="registry" containerID="cri-o://82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19" gracePeriod=30 Sep 09 12:16:38.585618 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:38.585436 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="offline" containerID="cri-o://66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635" gracePeriod=30 Sep 09 12:16:38.585618 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:38.585458 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="online" containerID="cri-o://910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33" gracePeriod=30 Sep 09 12:16:38.615197 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:38.615165 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-bk7gl"] Sep 09 12:16:38.615399 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:38.615353 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-bk7gl" podUID="fb85721e-f0d8-4863-baf1-470284c27d7d" containerName="redis" containerID="cri-o://afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f" gracePeriod=30 Sep 09 12:16:38.850345 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:38.850323 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-bk7gl" Sep 09 12:16:38.930694 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:38.930666 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9sp6\" (UniqueName: \"kubernetes.io/projected/fb85721e-f0d8-4863-baf1-470284c27d7d-kube-api-access-p9sp6\") pod \"fb85721e-f0d8-4863-baf1-470284c27d7d\" (UID: \"fb85721e-f0d8-4863-baf1-470284c27d7d\") " Sep 09 12:16:38.932704 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:38.932672 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb85721e-f0d8-4863-baf1-470284c27d7d-kube-api-access-p9sp6" (OuterVolumeSpecName: "kube-api-access-p9sp6") pod "fb85721e-f0d8-4863-baf1-470284c27d7d" (UID: "fb85721e-f0d8-4863-baf1-470284c27d7d"). InnerVolumeSpecName "kube-api-access-p9sp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:16:39.031889 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.031835 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9sp6\" (UniqueName: \"kubernetes.io/projected/fb85721e-f0d8-4863-baf1-470284c27d7d-kube-api-access-p9sp6\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:16:39.509049 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.509019 2556 generic.go:356] "Generic (PLEG): container finished" podID="fb85721e-f0d8-4863-baf1-470284c27d7d" containerID="afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f" exitCode=0 Sep 09 12:16:39.509168 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.509064 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-bk7gl" Sep 09 12:16:39.509168 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.509106 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-bk7gl" event={"ID":"fb85721e-f0d8-4863-baf1-470284c27d7d","Type":"ContainerDied","Data":"afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f"} Sep 09 12:16:39.509168 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.509150 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-bk7gl" event={"ID":"fb85721e-f0d8-4863-baf1-470284c27d7d","Type":"ContainerDied","Data":"47f06fdcdda2731b475853057be076b9f0b909b93f6df52bb26fb18317864b95"} Sep 09 12:16:39.509329 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.509168 2556 scope.go:117] "RemoveContainer" containerID="afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f" Sep 09 12:16:39.512090 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.512067 2556 generic.go:356] "Generic (PLEG): container finished" podID="bd059e10-664e-427a-b172-340443c18707" containerID="910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33" exitCode=0 Sep 09 12:16:39.512184 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.512096 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerDied","Data":"910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33"} Sep 09 12:16:39.516886 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.516869 2556 scope.go:117] "RemoveContainer" containerID="afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f" Sep 09 12:16:39.517152 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:16:39.517129 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f\": container with ID starting with afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f not found: ID does not exist" containerID="afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f" Sep 09 12:16:39.517277 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.517159 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f"} err="failed to get container status \"afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f\": rpc error: code = NotFound desc = could not find container \"afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f\": container with ID starting with afa429c992af4cb5f8b604649939f88ed4d8ed90200d98295ea98326f16f0a0f not found: ID does not exist" Sep 09 12:16:39.533968 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.533943 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-bk7gl"] Sep 09 12:16:39.538991 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:39.538973 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-bk7gl"] Sep 09 12:16:40.816463 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:40.816422 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb85721e-f0d8-4863-baf1-470284c27d7d" path="/var/lib/kubelet/pods/fb85721e-f0d8-4863-baf1-470284c27d7d/volumes" Sep 09 12:16:46.200291 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:46.200251 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 09 12:16:56.199846 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:16:56.199799 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 09 12:17:06.200588 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:06.200549 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 09 12:17:06.201033 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:06.200666 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:17:09.240848 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.240825 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:17:09.279573 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.279550 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bd059e10-664e-427a-b172-340443c18707-feast-data\") pod \"bd059e10-664e-427a-b172-340443c18707\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " Sep 09 12:17:09.279726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.279641 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgmxf\" (UniqueName: \"kubernetes.io/projected/bd059e10-664e-427a-b172-340443c18707-kube-api-access-lgmxf\") pod \"bd059e10-664e-427a-b172-340443c18707\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " Sep 09 12:17:09.279726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.279669 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-registry-tls\") pod \"bd059e10-664e-427a-b172-340443c18707\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " Sep 09 12:17:09.279726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.279684 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-online-tls\") pod \"bd059e10-664e-427a-b172-340443c18707\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " Sep 09 12:17:09.279726 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.279714 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-offline-tls\") pod \"bd059e10-664e-427a-b172-340443c18707\" (UID: \"bd059e10-664e-427a-b172-340443c18707\") " Sep 09 12:17:09.281677 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.281651 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd059e10-664e-427a-b172-340443c18707-kube-api-access-lgmxf" (OuterVolumeSpecName: "kube-api-access-lgmxf") pod "bd059e10-664e-427a-b172-340443c18707" (UID: "bd059e10-664e-427a-b172-340443c18707"). InnerVolumeSpecName "kube-api-access-lgmxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:17:09.282031 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.281994 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-online-tls" (OuterVolumeSpecName: "online-tls") pod "bd059e10-664e-427a-b172-340443c18707" (UID: "bd059e10-664e-427a-b172-340443c18707"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:17:09.282150 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.282100 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bd059e10-664e-427a-b172-340443c18707" (UID: "bd059e10-664e-427a-b172-340443c18707"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:17:09.282150 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.282117 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "bd059e10-664e-427a-b172-340443c18707" (UID: "bd059e10-664e-427a-b172-340443c18707"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:17:09.288467 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.288442 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd059e10-664e-427a-b172-340443c18707-feast-data" (OuterVolumeSpecName: "feast-data") pod "bd059e10-664e-427a-b172-340443c18707" (UID: "bd059e10-664e-427a-b172-340443c18707"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:17:09.380851 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.380788 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgmxf\" (UniqueName: \"kubernetes.io/projected/bd059e10-664e-427a-b172-340443c18707-kube-api-access-lgmxf\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:17:09.380851 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.380811 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-registry-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:17:09.380851 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.380822 2556 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-online-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:17:09.380851 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.380830 2556 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bd059e10-664e-427a-b172-340443c18707-offline-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:17:09.380851 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.380837 2556 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bd059e10-664e-427a-b172-340443c18707-feast-data\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:17:09.631571 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.631481 2556 generic.go:356] "Generic (PLEG): container finished" podID="bd059e10-664e-427a-b172-340443c18707" containerID="66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635" exitCode=137 Sep 09 12:17:09.631571 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.631505 2556 generic.go:356] "Generic (PLEG): container finished" podID="bd059e10-664e-427a-b172-340443c18707" containerID="82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19" exitCode=137 Sep 09 12:17:09.631571 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.631544 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" Sep 09 12:17:09.631852 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.631574 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerDied","Data":"66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635"} Sep 09 12:17:09.631852 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.631635 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerDied","Data":"82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19"} Sep 09 12:17:09.631852 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.631667 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx" event={"ID":"bd059e10-664e-427a-b172-340443c18707","Type":"ContainerDied","Data":"5430e238e7dbd0b2fe047a248c60d7ff9965bd4a8d3e4e08cb47409e25d9d3a5"} Sep 09 12:17:09.631852 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.631696 2556 scope.go:117] "RemoveContainer" containerID="66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635" Sep 09 12:17:09.641174 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.641158 2556 scope.go:117] "RemoveContainer" containerID="910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33" Sep 09 12:17:09.648450 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.648436 2556 scope.go:117] "RemoveContainer" containerID="82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19" Sep 09 12:17:09.655535 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.655515 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx"] Sep 09 12:17:09.656242 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.656227 2556 scope.go:117] "RemoveContainer" containerID="b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7" Sep 09 12:17:09.660018 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.659999 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-7d974d8755-h9hpx"] Sep 09 12:17:09.663286 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.663265 2556 scope.go:117] "RemoveContainer" containerID="3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe" Sep 09 12:17:09.669922 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.669907 2556 scope.go:117] "RemoveContainer" containerID="66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635" Sep 09 12:17:09.670186 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:17:09.670168 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635\": container with ID starting with 66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635 not found: ID does not exist" containerID="66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635" Sep 09 12:17:09.670256 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.670194 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635"} err="failed to get container status \"66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635\": rpc error: code = NotFound desc = could not find container \"66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635\": container with ID starting with 66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635 not found: ID does not exist" Sep 09 12:17:09.670256 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.670230 2556 scope.go:117] "RemoveContainer" containerID="910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33" Sep 09 12:17:09.670456 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:17:09.670440 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33\": container with ID starting with 910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33 not found: ID does not exist" containerID="910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33" Sep 09 12:17:09.670504 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.670461 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33"} err="failed to get container status \"910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33\": rpc error: code = NotFound desc = could not find container \"910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33\": container with ID starting with 910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33 not found: ID does not exist" Sep 09 12:17:09.670504 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.670475 2556 scope.go:117] "RemoveContainer" containerID="82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19" Sep 09 12:17:09.670663 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:17:09.670645 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19\": container with ID starting with 82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19 not found: ID does not exist" containerID="82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19" Sep 09 12:17:09.670749 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.670663 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19"} err="failed to get container status \"82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19\": rpc error: code = NotFound desc = could not find container \"82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19\": container with ID starting with 82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19 not found: ID does not exist" Sep 09 12:17:09.670749 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.670674 2556 scope.go:117] "RemoveContainer" containerID="b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7" Sep 09 12:17:09.670893 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:17:09.670877 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7\": container with ID starting with b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7 not found: ID does not exist" containerID="b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7" Sep 09 12:17:09.670938 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.670896 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7"} err="failed to get container status \"b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7\": rpc error: code = NotFound desc = could not find container \"b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7\": container with ID starting with b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7 not found: ID does not exist" Sep 09 12:17:09.670938 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.670907 2556 scope.go:117] "RemoveContainer" containerID="3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe" Sep 09 12:17:09.671101 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:17:09.671087 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe\": container with ID starting with 3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe not found: ID does not exist" containerID="3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe" Sep 09 12:17:09.671139 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.671103 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe"} err="failed to get container status \"3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe\": rpc error: code = NotFound desc = could not find container \"3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe\": container with ID starting with 3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe not found: ID does not exist" Sep 09 12:17:09.671139 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.671113 2556 scope.go:117] "RemoveContainer" containerID="66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635" Sep 09 12:17:09.671373 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.671350 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635"} err="failed to get container status \"66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635\": rpc error: code = NotFound desc = could not find container \"66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635\": container with ID starting with 66435e38dfee0e7c479991756be8199fe711aee13752416ff1d8631d9db56635 not found: ID does not exist" Sep 09 12:17:09.671425 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.671376 2556 scope.go:117] "RemoveContainer" containerID="910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33" Sep 09 12:17:09.671596 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.671581 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33"} err="failed to get container status \"910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33\": rpc error: code = NotFound desc = could not find container \"910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33\": container with ID starting with 910890a27efdbbd054599da96bd3081d024b225c3634e1b610f82f06afe23e33 not found: ID does not exist" Sep 09 12:17:09.671635 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.671596 2556 scope.go:117] "RemoveContainer" containerID="82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19" Sep 09 12:17:09.671786 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.671771 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19"} err="failed to get container status \"82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19\": rpc error: code = NotFound desc = could not find container \"82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19\": container with ID starting with 82ef0d73e65a9657e94900849f0da81383e656e157d81a629a419f31593b1f19 not found: ID does not exist" Sep 09 12:17:09.671786 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.671785 2556 scope.go:117] "RemoveContainer" containerID="b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7" Sep 09 12:17:09.671958 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.671945 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7"} err="failed to get container status \"b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7\": rpc error: code = NotFound desc = could not find container \"b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7\": container with ID starting with b5b5215750d091b7dd90b89e7ae7b65ded8d0819e0ac67a7c3c60be9bbe4e2c7 not found: ID does not exist" Sep 09 12:17:09.671997 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.671958 2556 scope.go:117] "RemoveContainer" containerID="3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe" Sep 09 12:17:09.672180 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:09.672163 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe"} err="failed to get container status \"3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe\": rpc error: code = NotFound desc = could not find container \"3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe\": container with ID starting with 3392064d9ce14d1bf14ed948ceff9f0c0097aa1ba8a00a31b492b2a7fb8500fe not found: ID does not exist" Sep 09 12:17:10.815776 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:17:10.815744 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd059e10-664e-427a-b172-340443c18707" path="/var/lib/kubelet/pods/bd059e10-664e-427a-b172-340443c18707/volumes" Sep 09 12:19:13.188399 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188368 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-xq5f7"] Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188713 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb85721e-f0d8-4863-baf1-470284c27d7d" containerName="redis" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188726 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb85721e-f0d8-4863-baf1-470284c27d7d" containerName="redis" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188739 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="feast-init" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188747 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="feast-init" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188757 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="offline" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188763 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="offline" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188776 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="registry" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188781 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="registry" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188789 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="online" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188794 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="online" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188801 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="feast-apply" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188806 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="feast-apply" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188857 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="offline" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188867 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="online" Sep 09 12:19:13.188867 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188874 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd059e10-664e-427a-b172-340443c18707" containerName="registry" Sep 09 12:19:13.189341 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.188880 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb85721e-f0d8-4863-baf1-470284c27d7d" containerName="redis" Sep 09 12:19:13.191771 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.191756 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" Sep 09 12:19:13.194751 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.194729 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 09 12:19:13.194873 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.194796 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 09 12:19:13.194873 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.194846 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-s7666\"" Sep 09 12:19:13.205999 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.205976 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-xq5f7"] Sep 09 12:19:13.270431 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.270403 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65g98\" (UniqueName: \"kubernetes.io/projected/436bf2d4-b3e6-4be0-a16f-00c7b5ddd248-kube-api-access-65g98\") pod \"redis-69bbd798b6-xq5f7\" (UID: \"436bf2d4-b3e6-4be0-a16f-00c7b5ddd248\") " pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" Sep 09 12:19:13.371716 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.371666 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65g98\" (UniqueName: \"kubernetes.io/projected/436bf2d4-b3e6-4be0-a16f-00c7b5ddd248-kube-api-access-65g98\") pod \"redis-69bbd798b6-xq5f7\" (UID: \"436bf2d4-b3e6-4be0-a16f-00c7b5ddd248\") " pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" Sep 09 12:19:13.380458 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.380434 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65g98\" (UniqueName: \"kubernetes.io/projected/436bf2d4-b3e6-4be0-a16f-00c7b5ddd248-kube-api-access-65g98\") pod \"redis-69bbd798b6-xq5f7\" (UID: \"436bf2d4-b3e6-4be0-a16f-00c7b5ddd248\") " pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" Sep 09 12:19:13.507570 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.507469 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" Sep 09 12:19:13.777439 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:13.775935 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-xq5f7"] Sep 09 12:19:13.778300 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:19:13.778273 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod436bf2d4_b3e6_4be0_a16f_00c7b5ddd248.slice/crio-fdbe5ffb7b6acf069e41c4ecbc89762b40f0bdfb3fd6949412ce274fc86b2d80 WatchSource:0}: Error finding container fdbe5ffb7b6acf069e41c4ecbc89762b40f0bdfb3fd6949412ce274fc86b2d80: Status 404 returned error can't find the container with id fdbe5ffb7b6acf069e41c4ecbc89762b40f0bdfb3fd6949412ce274fc86b2d80 Sep 09 12:19:14.106395 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:14.106315 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" event={"ID":"436bf2d4-b3e6-4be0-a16f-00c7b5ddd248","Type":"ContainerStarted","Data":"5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956"} Sep 09 12:19:14.106395 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:14.106353 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" event={"ID":"436bf2d4-b3e6-4be0-a16f-00c7b5ddd248","Type":"ContainerStarted","Data":"fdbe5ffb7b6acf069e41c4ecbc89762b40f0bdfb3fd6949412ce274fc86b2d80"} Sep 09 12:19:14.121854 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:14.121811 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" podStartSLOduration=1.12179528 podStartE2EDuration="1.12179528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:19:14.121373362 +0000 UTC m=+1009.801279562" watchObservedRunningTime="2026-09-09 12:19:14.12179528 +0000 UTC m=+1009.801701468" Sep 09 12:19:24.208285 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.208252 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh"] Sep 09 12:19:24.211865 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.211848 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.214487 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.214455 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 09 12:19:24.214618 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.214543 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 09 12:19:24.214618 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.214546 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-f5wfm\"" Sep 09 12:19:24.214744 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.214635 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 09 12:19:24.214788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.214779 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 09 12:19:24.225954 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.225934 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh"] Sep 09 12:19:24.257876 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.257848 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-registry-tls\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.258067 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.257884 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-feast-data\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.258067 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.257906 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-online-tls\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.258067 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.257925 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5577g\" (UniqueName: \"kubernetes.io/projected/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-kube-api-access-5577g\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.359289 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.359255 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-online-tls\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.359289 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.359289 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5577g\" (UniqueName: \"kubernetes.io/projected/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-kube-api-access-5577g\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.359495 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.359355 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-registry-tls\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.359495 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.359376 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-feast-data\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.359756 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.359733 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-feast-data\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.361783 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.361752 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-online-tls\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.361871 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.361801 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-registry-tls\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.368183 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.368158 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5577g\" (UniqueName: \"kubernetes.io/projected/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-kube-api-access-5577g\") pod \"feast-credit-scoring-8c4546dc7-txwkh\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.520933 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.520842 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:24.795865 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:24.795786 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh"] Sep 09 12:19:24.799065 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:19:24.799033 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe230c4_d8d4_4cad_b8ef_100eeb690d9b.slice/crio-d492bffc9625221304081278ce0c2989282150b5a2ac75f5010c623d31728a23 WatchSource:0}: Error finding container d492bffc9625221304081278ce0c2989282150b5a2ac75f5010c623d31728a23: Status 404 returned error can't find the container with id d492bffc9625221304081278ce0c2989282150b5a2ac75f5010c623d31728a23 Sep 09 12:19:25.152723 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:25.152636 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" event={"ID":"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b","Type":"ContainerStarted","Data":"f6a022d0ad39a1f2d5ddb8f454724a2d15e541d0217e09558955a2452e03f996"} Sep 09 12:19:25.152723 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:25.152714 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" event={"ID":"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b","Type":"ContainerStarted","Data":"d492bffc9625221304081278ce0c2989282150b5a2ac75f5010c623d31728a23"} Sep 09 12:19:27.162625 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:27.162589 2556 generic.go:356] "Generic (PLEG): container finished" podID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerID="f6a022d0ad39a1f2d5ddb8f454724a2d15e541d0217e09558955a2452e03f996" exitCode=0 Sep 09 12:19:27.162962 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:27.162664 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" event={"ID":"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b","Type":"ContainerDied","Data":"f6a022d0ad39a1f2d5ddb8f454724a2d15e541d0217e09558955a2452e03f996"} Sep 09 12:19:27.168400 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:27.168382 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 09 12:19:28.169042 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:28.169010 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" event={"ID":"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b","Type":"ContainerStarted","Data":"931113220be475383b634ceb64a86f16c3dddf5fdfb3076c04ce700152434e66"} Sep 09 12:19:35.201812 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:35.201775 2556 generic.go:356] "Generic (PLEG): container finished" podID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerID="931113220be475383b634ceb64a86f16c3dddf5fdfb3076c04ce700152434e66" exitCode=0 Sep 09 12:19:35.202276 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:35.201847 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" event={"ID":"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b","Type":"ContainerDied","Data":"931113220be475383b634ceb64a86f16c3dddf5fdfb3076c04ce700152434e66"} Sep 09 12:19:35.205708 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:35.205689 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 09 12:19:36.209800 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:36.209770 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" event={"ID":"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b","Type":"ContainerStarted","Data":"c1f9a74d1fe94f8f5ad8de03be537b6ea3563b9443ca11b1b5440925a6faa395"} Sep 09 12:19:36.210171 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:36.209823 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" event={"ID":"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b","Type":"ContainerStarted","Data":"1d8fa6775d5bc18fbb22371046f7bd6df6e27d71dcab816dc5873fb6ac7d5b2b"} Sep 09 12:19:36.235373 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:36.235322 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podStartSLOduration=12.235304946 podStartE2EDuration="12.235304946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:19:36.235262662 +0000 UTC m=+1031.915168850" watchObservedRunningTime="2026-09-09 12:19:36.235304946 +0000 UTC m=+1031.915211133" Sep 09 12:19:36.521789 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:36.521694 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:36.521789 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:36.521756 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:36.523127 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:36.523084 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 09 12:19:36.523127 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:36.523117 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 12:19:39.522016 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:39.521970 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 09 12:19:39.522441 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:39.521981 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 12:19:42.522087 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:42.522040 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 09 12:19:42.527864 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:42.527840 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:42.528331 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:42.528312 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:42.531315 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:42.531294 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:44.521940 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:44.521898 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:45.521232 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:45.521178 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 09 12:19:48.521875 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:48.521838 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:48.522635 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:48.522614 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:19:50.033309 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.033275 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p"] Sep 09 12:19:50.037108 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.037090 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.039680 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.039649 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-x5dsr\"" Sep 09 12:19:50.039803 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.039650 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 09 12:19:50.039803 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.039707 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 09 12:19:50.039893 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.039803 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 09 12:19:50.047468 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.047448 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p"] Sep 09 12:19:50.207533 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.207499 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/656e2bed-03f2-4078-a9be-072c6c62a934-feast-data\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.207686 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.207549 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6fmp\" (UniqueName: \"kubernetes.io/projected/656e2bed-03f2-4078-a9be-072c6c62a934-kube-api-access-j6fmp\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.207686 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.207601 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-online-tls\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.207686 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.207637 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-registry-tls\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.308435 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.308360 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/656e2bed-03f2-4078-a9be-072c6c62a934-feast-data\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.308435 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.308404 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6fmp\" (UniqueName: \"kubernetes.io/projected/656e2bed-03f2-4078-a9be-072c6c62a934-kube-api-access-j6fmp\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.308435 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.308425 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-online-tls\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.308611 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.308450 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-registry-tls\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.308611 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:19:50.308555 2556 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found Sep 09 12:19:50.308680 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:19:50.308624 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-registry-tls podName:656e2bed-03f2-4078-a9be-072c6c62a934 nodeName:}" failed. No retries permitted until 2026-09-09 12:19:50.808605162 +0000 UTC m=+1046.488511327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-registry-tls") pod "feast-driver-ranking-5ccdd6f8f4-dv22p" (UID: "656e2bed-03f2-4078-a9be-072c6c62a934") : secret "feast-driver-ranking-registry-tls" not found Sep 09 12:19:50.308831 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.308810 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/656e2bed-03f2-4078-a9be-072c6c62a934-feast-data\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.310784 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.310757 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-online-tls\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.318138 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.318116 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6fmp\" (UniqueName: \"kubernetes.io/projected/656e2bed-03f2-4078-a9be-072c6c62a934-kube-api-access-j6fmp\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.813060 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.813027 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-registry-tls\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.815591 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.815563 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-registry-tls\") pod \"feast-driver-ranking-5ccdd6f8f4-dv22p\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:50.947360 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:50.947333 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:19:51.224500 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:51.224476 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p"] Sep 09 12:19:51.226846 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:19:51.226818 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656e2bed_03f2_4078_a9be_072c6c62a934.slice/crio-48eb42fec342300e004868ca1a45a920407240a0be489a8ea3434c8e030a3f8a WatchSource:0}: Error finding container 48eb42fec342300e004868ca1a45a920407240a0be489a8ea3434c8e030a3f8a: Status 404 returned error can't find the container with id 48eb42fec342300e004868ca1a45a920407240a0be489a8ea3434c8e030a3f8a Sep 09 12:19:51.280924 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:51.280896 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" event={"ID":"656e2bed-03f2-4078-a9be-072c6c62a934","Type":"ContainerStarted","Data":"48eb42fec342300e004868ca1a45a920407240a0be489a8ea3434c8e030a3f8a"} Sep 09 12:19:52.286843 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:52.286808 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" event={"ID":"656e2bed-03f2-4078-a9be-072c6c62a934","Type":"ContainerStarted","Data":"a69fb3eee5cdc6c6049a992b810220902a166a56d1151cd8af600b187274c296"} Sep 09 12:19:56.308611 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:56.308531 2556 generic.go:356] "Generic (PLEG): container finished" podID="656e2bed-03f2-4078-a9be-072c6c62a934" containerID="a69fb3eee5cdc6c6049a992b810220902a166a56d1151cd8af600b187274c296" exitCode=0 Sep 09 12:19:56.308987 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:56.308608 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" event={"ID":"656e2bed-03f2-4078-a9be-072c6c62a934","Type":"ContainerDied","Data":"a69fb3eee5cdc6c6049a992b810220902a166a56d1151cd8af600b187274c296"} Sep 09 12:19:57.314795 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:19:57.314756 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" event={"ID":"656e2bed-03f2-4078-a9be-072c6c62a934","Type":"ContainerStarted","Data":"ab17f8b2067c38c3bda3c7ae3415b768e4c8ff2528e95c51e3de9556eeaba5de"} Sep 09 12:20:04.347623 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:04.347588 2556 generic.go:356] "Generic (PLEG): container finished" podID="656e2bed-03f2-4078-a9be-072c6c62a934" containerID="ab17f8b2067c38c3bda3c7ae3415b768e4c8ff2528e95c51e3de9556eeaba5de" exitCode=0 Sep 09 12:20:04.348000 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:04.347643 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" event={"ID":"656e2bed-03f2-4078-a9be-072c6c62a934","Type":"ContainerDied","Data":"ab17f8b2067c38c3bda3c7ae3415b768e4c8ff2528e95c51e3de9556eeaba5de"} Sep 09 12:20:05.355778 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:05.355748 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" event={"ID":"656e2bed-03f2-4078-a9be-072c6c62a934","Type":"ContainerStarted","Data":"ada4b055899d8af237cd8732c982da0867d6de6634a18d6fbc4d5f705e79b27c"} Sep 09 12:20:05.356130 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:05.355798 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" event={"ID":"656e2bed-03f2-4078-a9be-072c6c62a934","Type":"ContainerStarted","Data":"8e80f6fa12851568afdc08bf1d1f48a2e8e873a9dab3ffad5ded6eac322aecd6"} Sep 09 12:20:05.380386 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:05.380324 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podStartSLOduration=15.380306037 podStartE2EDuration="15.380306037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:20:05.377709789 +0000 UTC m=+1061.057615981" watchObservedRunningTime="2026-09-09 12:20:05.380306037 +0000 UTC m=+1061.060212225" Sep 09 12:20:05.948167 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:05.948120 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:20:05.948355 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:05.948230 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:20:05.949474 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:05.949443 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 09 12:20:05.949579 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:05.949491 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 09 12:20:08.948030 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:08.947989 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 09 12:20:08.948445 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:08.948000 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 09 12:20:10.947768 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:10.947719 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:20:10.948261 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:10.947818 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:20:11.948429 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:11.948387 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 09 12:20:11.954524 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:11.954497 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:20:12.395854 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:12.395766 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:20:14.948064 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:14.948020 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 09 12:20:17.949118 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:17.949040 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:20:17.949779 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:20:17.949762 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:21:38.825025 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:38.824994 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh"] Sep 09 12:21:38.825755 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:38.825222 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="registry" containerID="cri-o://1d8fa6775d5bc18fbb22371046f7bd6df6e27d71dcab816dc5873fb6ac7d5b2b" gracePeriod=30 Sep 09 12:21:38.825755 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:38.825264 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="online" containerID="cri-o://c1f9a74d1fe94f8f5ad8de03be537b6ea3563b9443ca11b1b5440925a6faa395" gracePeriod=30 Sep 09 12:21:38.834446 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:38.834417 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p"] Sep 09 12:21:38.834635 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:38.834611 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="registry" containerID="cri-o://8e80f6fa12851568afdc08bf1d1f48a2e8e873a9dab3ffad5ded6eac322aecd6" gracePeriod=30 Sep 09 12:21:38.834748 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:38.834631 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="online" containerID="cri-o://ada4b055899d8af237cd8732c982da0867d6de6634a18d6fbc4d5f705e79b27c" gracePeriod=30 Sep 09 12:21:39.768761 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:39.768728 2556 generic.go:356] "Generic (PLEG): container finished" podID="656e2bed-03f2-4078-a9be-072c6c62a934" containerID="ada4b055899d8af237cd8732c982da0867d6de6634a18d6fbc4d5f705e79b27c" exitCode=0 Sep 09 12:21:39.768932 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:39.768803 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" event={"ID":"656e2bed-03f2-4078-a9be-072c6c62a934","Type":"ContainerDied","Data":"ada4b055899d8af237cd8732c982da0867d6de6634a18d6fbc4d5f705e79b27c"} Sep 09 12:21:39.771139 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:39.771117 2556 generic.go:356] "Generic (PLEG): container finished" podID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerID="c1f9a74d1fe94f8f5ad8de03be537b6ea3563b9443ca11b1b5440925a6faa395" exitCode=0 Sep 09 12:21:39.771266 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:39.771145 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" event={"ID":"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b","Type":"ContainerDied","Data":"c1f9a74d1fe94f8f5ad8de03be537b6ea3563b9443ca11b1b5440925a6faa395"} Sep 09 12:21:41.222763 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.222728 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-xq5f7"] Sep 09 12:21:41.223169 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.222925 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" podUID="436bf2d4-b3e6-4be0-a16f-00c7b5ddd248" containerName="redis" containerID="cri-o://5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956" gracePeriod=30 Sep 09 12:21:41.454739 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.454717 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" Sep 09 12:21:41.565166 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.565089 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65g98\" (UniqueName: \"kubernetes.io/projected/436bf2d4-b3e6-4be0-a16f-00c7b5ddd248-kube-api-access-65g98\") pod \"436bf2d4-b3e6-4be0-a16f-00c7b5ddd248\" (UID: \"436bf2d4-b3e6-4be0-a16f-00c7b5ddd248\") " Sep 09 12:21:41.567140 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.567115 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436bf2d4-b3e6-4be0-a16f-00c7b5ddd248-kube-api-access-65g98" (OuterVolumeSpecName: "kube-api-access-65g98") pod "436bf2d4-b3e6-4be0-a16f-00c7b5ddd248" (UID: "436bf2d4-b3e6-4be0-a16f-00c7b5ddd248"). InnerVolumeSpecName "kube-api-access-65g98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:21:41.666279 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.666250 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65g98\" (UniqueName: \"kubernetes.io/projected/436bf2d4-b3e6-4be0-a16f-00c7b5ddd248-kube-api-access-65g98\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:21:41.781199 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.781170 2556 generic.go:356] "Generic (PLEG): container finished" podID="436bf2d4-b3e6-4be0-a16f-00c7b5ddd248" containerID="5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956" exitCode=0 Sep 09 12:21:41.781377 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.781229 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" Sep 09 12:21:41.781377 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.781230 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" event={"ID":"436bf2d4-b3e6-4be0-a16f-00c7b5ddd248","Type":"ContainerDied","Data":"5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956"} Sep 09 12:21:41.781377 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.781273 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-xq5f7" event={"ID":"436bf2d4-b3e6-4be0-a16f-00c7b5ddd248","Type":"ContainerDied","Data":"fdbe5ffb7b6acf069e41c4ecbc89762b40f0bdfb3fd6949412ce274fc86b2d80"} Sep 09 12:21:41.781377 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.781291 2556 scope.go:117] "RemoveContainer" containerID="5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956" Sep 09 12:21:41.789404 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.789389 2556 scope.go:117] "RemoveContainer" containerID="5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956" Sep 09 12:21:41.789650 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:21:41.789631 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956\": container with ID starting with 5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956 not found: ID does not exist" containerID="5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956" Sep 09 12:21:41.789703 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.789659 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956"} err="failed to get container status \"5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956\": rpc error: code = NotFound desc = could not find container \"5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956\": container with ID starting with 5471cc9a7389b1aa5393a5b3c639b52c0b31bca2d727d499f1edaadb74a7f956 not found: ID does not exist" Sep 09 12:21:41.802397 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.802378 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-xq5f7"] Sep 09 12:21:41.806682 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:41.806664 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-xq5f7"] Sep 09 12:21:42.392089 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:42.392054 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 09 12:21:42.528877 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:42.528840 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 12:21:42.816141 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:42.816106 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="436bf2d4-b3e6-4be0-a16f-00c7b5ddd248" path="/var/lib/kubelet/pods/436bf2d4-b3e6-4be0-a16f-00c7b5ddd248/volumes" Sep 09 12:21:52.392544 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:52.392505 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 09 12:21:52.529140 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:21:52.529105 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 12:22:02.392067 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:02.392021 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 09 12:22:02.392471 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:02.392122 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:22:02.528801 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:02.528760 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 12:22:02.528992 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:02.528879 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:22:08.894668 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:08.894642 2556 generic.go:356] "Generic (PLEG): container finished" podID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerID="1d8fa6775d5bc18fbb22371046f7bd6df6e27d71dcab816dc5873fb6ac7d5b2b" exitCode=137 Sep 09 12:22:08.894979 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:08.894716 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" event={"ID":"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b","Type":"ContainerDied","Data":"1d8fa6775d5bc18fbb22371046f7bd6df6e27d71dcab816dc5873fb6ac7d5b2b"} Sep 09 12:22:08.897048 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:08.897028 2556 generic.go:356] "Generic (PLEG): container finished" podID="656e2bed-03f2-4078-a9be-072c6c62a934" containerID="8e80f6fa12851568afdc08bf1d1f48a2e8e873a9dab3ffad5ded6eac322aecd6" exitCode=137 Sep 09 12:22:08.897134 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:08.897072 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" event={"ID":"656e2bed-03f2-4078-a9be-072c6c62a934","Type":"ContainerDied","Data":"8e80f6fa12851568afdc08bf1d1f48a2e8e873a9dab3ffad5ded6eac322aecd6"} Sep 09 12:22:09.484502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.484481 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:22:09.487011 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.486995 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:22:09.597340 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.597261 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/656e2bed-03f2-4078-a9be-072c6c62a934-feast-data\") pod \"656e2bed-03f2-4078-a9be-072c6c62a934\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " Sep 09 12:22:09.597340 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.597312 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5577g\" (UniqueName: \"kubernetes.io/projected/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-kube-api-access-5577g\") pod \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " Sep 09 12:22:09.597558 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.597347 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-feast-data\") pod \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " Sep 09 12:22:09.597558 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.597378 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-registry-tls\") pod \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " Sep 09 12:22:09.597558 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.597419 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-online-tls\") pod \"656e2bed-03f2-4078-a9be-072c6c62a934\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " Sep 09 12:22:09.597558 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.597452 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6fmp\" (UniqueName: \"kubernetes.io/projected/656e2bed-03f2-4078-a9be-072c6c62a934-kube-api-access-j6fmp\") pod \"656e2bed-03f2-4078-a9be-072c6c62a934\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " Sep 09 12:22:09.597558 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.597470 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-online-tls\") pod \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\" (UID: \"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b\") " Sep 09 12:22:09.597558 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.597492 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-registry-tls\") pod \"656e2bed-03f2-4078-a9be-072c6c62a934\" (UID: \"656e2bed-03f2-4078-a9be-072c6c62a934\") " Sep 09 12:22:09.597873 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.597839 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/656e2bed-03f2-4078-a9be-072c6c62a934-feast-data" (OuterVolumeSpecName: "feast-data") pod "656e2bed-03f2-4078-a9be-072c6c62a934" (UID: "656e2bed-03f2-4078-a9be-072c6c62a934"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:22:09.599641 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.599604 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-online-tls" (OuterVolumeSpecName: "online-tls") pod "656e2bed-03f2-4078-a9be-072c6c62a934" (UID: "656e2bed-03f2-4078-a9be-072c6c62a934"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:22:09.599985 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.599949 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" (UID: "5fe230c4-d8d4-4cad-b8ef-100eeb690d9b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:22:09.600088 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.600002 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-online-tls" (OuterVolumeSpecName: "online-tls") pod "5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" (UID: "5fe230c4-d8d4-4cad-b8ef-100eeb690d9b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:22:09.600088 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.600050 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-kube-api-access-5577g" (OuterVolumeSpecName: "kube-api-access-5577g") pod "5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" (UID: "5fe230c4-d8d4-4cad-b8ef-100eeb690d9b"). InnerVolumeSpecName "kube-api-access-5577g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:22:09.600088 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.600062 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656e2bed-03f2-4078-a9be-072c6c62a934-kube-api-access-j6fmp" (OuterVolumeSpecName: "kube-api-access-j6fmp") pod "656e2bed-03f2-4078-a9be-072c6c62a934" (UID: "656e2bed-03f2-4078-a9be-072c6c62a934"). InnerVolumeSpecName "kube-api-access-j6fmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:22:09.600286 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.600247 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "656e2bed-03f2-4078-a9be-072c6c62a934" (UID: "656e2bed-03f2-4078-a9be-072c6c62a934"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:22:09.607497 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.607475 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-feast-data" (OuterVolumeSpecName: "feast-data") pod "5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" (UID: "5fe230c4-d8d4-4cad-b8ef-100eeb690d9b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:22:09.698431 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.698410 2556 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/656e2bed-03f2-4078-a9be-072c6c62a934-feast-data\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:22:09.698431 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.698430 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5577g\" (UniqueName: \"kubernetes.io/projected/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-kube-api-access-5577g\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:22:09.698614 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.698440 2556 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-feast-data\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:22:09.698614 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.698449 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-registry-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:22:09.698614 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.698474 2556 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-online-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:22:09.698614 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.698483 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6fmp\" (UniqueName: \"kubernetes.io/projected/656e2bed-03f2-4078-a9be-072c6c62a934-kube-api-access-j6fmp\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:22:09.698614 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.698491 2556 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b-online-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:22:09.698614 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.698498 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/656e2bed-03f2-4078-a9be-072c6c62a934-registry-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:22:09.903247 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.903145 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" Sep 09 12:22:09.903247 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.903165 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh" event={"ID":"5fe230c4-d8d4-4cad-b8ef-100eeb690d9b","Type":"ContainerDied","Data":"d492bffc9625221304081278ce0c2989282150b5a2ac75f5010c623d31728a23"} Sep 09 12:22:09.903247 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.903244 2556 scope.go:117] "RemoveContainer" containerID="c1f9a74d1fe94f8f5ad8de03be537b6ea3563b9443ca11b1b5440925a6faa395" Sep 09 12:22:09.905511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.905494 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" Sep 09 12:22:09.905624 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.905493 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p" event={"ID":"656e2bed-03f2-4078-a9be-072c6c62a934","Type":"ContainerDied","Data":"48eb42fec342300e004868ca1a45a920407240a0be489a8ea3434c8e030a3f8a"} Sep 09 12:22:09.912921 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.912898 2556 scope.go:117] "RemoveContainer" containerID="1d8fa6775d5bc18fbb22371046f7bd6df6e27d71dcab816dc5873fb6ac7d5b2b" Sep 09 12:22:09.920545 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.920528 2556 scope.go:117] "RemoveContainer" containerID="931113220be475383b634ceb64a86f16c3dddf5fdfb3076c04ce700152434e66" Sep 09 12:22:09.926803 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.926784 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh"] Sep 09 12:22:09.928065 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.928049 2556 scope.go:117] "RemoveContainer" containerID="f6a022d0ad39a1f2d5ddb8f454724a2d15e541d0217e09558955a2452e03f996" Sep 09 12:22:09.929627 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.929608 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-8c4546dc7-txwkh"] Sep 09 12:22:09.935037 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.935023 2556 scope.go:117] "RemoveContainer" containerID="ada4b055899d8af237cd8732c982da0867d6de6634a18d6fbc4d5f705e79b27c" Sep 09 12:22:09.941322 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.941302 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p"] Sep 09 12:22:09.942438 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.942421 2556 scope.go:117] "RemoveContainer" containerID="8e80f6fa12851568afdc08bf1d1f48a2e8e873a9dab3ffad5ded6eac322aecd6" Sep 09 12:22:09.946293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.946271 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5ccdd6f8f4-dv22p"] Sep 09 12:22:09.950416 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.950400 2556 scope.go:117] "RemoveContainer" containerID="ab17f8b2067c38c3bda3c7ae3415b768e4c8ff2528e95c51e3de9556eeaba5de" Sep 09 12:22:09.957071 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:09.957052 2556 scope.go:117] "RemoveContainer" containerID="a69fb3eee5cdc6c6049a992b810220902a166a56d1151cd8af600b187274c296" Sep 09 12:22:10.815885 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:10.815853 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" path="/var/lib/kubelet/pods/5fe230c4-d8d4-4cad-b8ef-100eeb690d9b/volumes" Sep 09 12:22:10.816367 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:10.816354 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" path="/var/lib/kubelet/pods/656e2bed-03f2-4078-a9be-072c6c62a934/volumes" Sep 09 12:22:49.725291 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725256 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj"] Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725580 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="feast-apply" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725592 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="feast-apply" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725601 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="online" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725606 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="online" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725614 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="436bf2d4-b3e6-4be0-a16f-00c7b5ddd248" containerName="redis" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725620 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="436bf2d4-b3e6-4be0-a16f-00c7b5ddd248" containerName="redis" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725629 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="feast-apply" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725634 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="feast-apply" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725642 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="feast-init" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725647 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="feast-init" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725657 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="online" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725662 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="online" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725668 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="registry" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725673 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="registry" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725682 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="feast-init" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725687 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="feast-init" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725694 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="registry" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725700 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="registry" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725747 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="online" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725755 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="436bf2d4-b3e6-4be0-a16f-00c7b5ddd248" containerName="redis" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725764 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="online" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725770 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe230c4-d8d4-4cad-b8ef-100eeb690d9b" containerName="registry" Sep 09 12:22:49.725798 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.725776 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="656e2bed-03f2-4078-a9be-072c6c62a934" containerName="registry" Sep 09 12:22:49.730598 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.730579 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.733332 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.733311 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 12:22:49.733461 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.733313 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 12:22:49.733461 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.733437 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 12:22:49.733652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.733636 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 12:22:49.733727 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.733685 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 12:22:49.733727 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.733703 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 12:22:49.733827 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.733787 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-bgrc9\"" Sep 09 12:22:49.742092 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.742071 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj"] Sep 09 12:22:49.815130 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.815103 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.815292 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.815138 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.815292 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.815179 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.815292 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.815245 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.815292 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.815280 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/adddf804-fcf8-41e6-bac0-1bf1d779837e-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.815455 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.815323 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqw5w\" (UniqueName: \"kubernetes.io/projected/adddf804-fcf8-41e6-bac0-1bf1d779837e-kube-api-access-sqw5w\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.915985 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.915951 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqw5w\" (UniqueName: \"kubernetes.io/projected/adddf804-fcf8-41e6-bac0-1bf1d779837e-kube-api-access-sqw5w\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.916175 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.916054 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.916175 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.916083 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.916391 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.916363 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.916500 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.916480 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.916781 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.916758 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/adddf804-fcf8-41e6-bac0-1bf1d779837e-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.917220 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.917179 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/adddf804-fcf8-41e6-bac0-1bf1d779837e-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.918718 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.918695 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.918862 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.918842 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.919124 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.919105 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.919198 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.919105 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:49.926555 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:49.926527 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqw5w\" (UniqueName: \"kubernetes.io/projected/adddf804-fcf8-41e6-bac0-1bf1d779837e-kube-api-access-sqw5w\") pod \"feast-simple-feast-setup-57576b8fcf-qhnpj\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:50.049016 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:50.048940 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:22:50.324293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:50.324267 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj"] Sep 09 12:22:50.326579 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:22:50.326553 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadddf804_fcf8_41e6_bac0_1bf1d779837e.slice/crio-15ea6a92b7f4a2c0ff85c2cffd6832e361318a8c94e938f25f5824e4e159b372 WatchSource:0}: Error finding container 15ea6a92b7f4a2c0ff85c2cffd6832e361318a8c94e938f25f5824e4e159b372: Status 404 returned error can't find the container with id 15ea6a92b7f4a2c0ff85c2cffd6832e361318a8c94e938f25f5824e4e159b372 Sep 09 12:22:50.328771 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:50.328754 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 12:22:51.069641 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:22:51.069610 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerStarted","Data":"15ea6a92b7f4a2c0ff85c2cffd6832e361318a8c94e938f25f5824e4e159b372"} Sep 09 12:23:05.142139 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:05.142097 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerStarted","Data":"732c11d8f89bdfd1cc2a704ea968be84fcd1e6aeb0b4a9ceb0b74b12f06cedf2"} Sep 09 12:23:09.160652 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:09.160619 2556 generic.go:356] "Generic (PLEG): container finished" podID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerID="732c11d8f89bdfd1cc2a704ea968be84fcd1e6aeb0b4a9ceb0b74b12f06cedf2" exitCode=0 Sep 09 12:23:09.161002 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:09.160681 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerDied","Data":"732c11d8f89bdfd1cc2a704ea968be84fcd1e6aeb0b4a9ceb0b74b12f06cedf2"} Sep 09 12:23:10.165949 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:10.165914 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerStarted","Data":"846974733d631b7185f9b04ff050a782e7af34818e6f137ab7ff390831c05159"} Sep 09 12:23:15.188896 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:15.188864 2556 generic.go:356] "Generic (PLEG): container finished" podID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerID="846974733d631b7185f9b04ff050a782e7af34818e6f137ab7ff390831c05159" exitCode=0 Sep 09 12:23:15.189297 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:15.188942 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerDied","Data":"846974733d631b7185f9b04ff050a782e7af34818e6f137ab7ff390831c05159"} Sep 09 12:23:16.198954 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:16.198908 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerStarted","Data":"74dd162114b5e8026d8e7506bf33698440eaa232c7462e084408ca19e4611c3c"} Sep 09 12:23:16.199462 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:16.198973 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerStarted","Data":"b2801bc9846a229744a2339b4e3848fac865c9fdda4fe430859a3d3a64409ef3"} Sep 09 12:23:16.199462 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:16.198994 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerStarted","Data":"f047ef4b46c2a9c922271c81fa184b1f5c68ccf897997de5fd4624f1e13c5e8c"} Sep 09 12:23:16.199462 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:16.199012 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerStarted","Data":"7f707df0d0c7956f897746b384ed83c763c38678eeb884239a3700dced7b99e2"} Sep 09 12:23:16.228391 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:16.228320 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podStartSLOduration=13.410941005 podStartE2EDuration="27.228298473s" totalImagesPullingTime="13.817357468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:23:16.225883545 +0000 UTC m=+1251.905789725" watchObservedRunningTime="2026-09-09 12:23:16.228298473 +0000 UTC m=+1251.908204660" Sep 09 12:23:17.050344 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:17.050133 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:17.050344 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:17.050193 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:17.050344 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:17.050236 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:17.050344 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:17.050256 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:17.052343 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:17.052285 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused" Sep 09 12:23:17.052343 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:17.052333 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Sep 09 12:23:17.052529 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:17.052304 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Sep 09 12:23:17.052529 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:17.052473 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 09 12:23:20.049951 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:20.049500 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:20.049951 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:20.049572 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:20.049951 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:20.049595 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:20.049951 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:20.049738 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused" Sep 09 12:23:20.049951 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:20.049757 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Sep 09 12:23:20.049951 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:20.049836 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 09 12:23:20.050899 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:20.050189 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:20.050899 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:20.050291 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Sep 09 12:23:23.050168 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:23.050124 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Sep 09 12:23:23.050765 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:23.050168 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Sep 09 12:23:23.051062 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:23.051042 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:23.051973 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:23.051950 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:23.086763 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:23.086736 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:23.243908 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:23.243881 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:26.050444 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:26.050414 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:26.050844 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:26.050562 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:26.253163 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:26.253132 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:23:26.253381 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:23:26.253363 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:24:28.339084 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.339051 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj"] Sep 09 12:24:28.340544 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.339315 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="registry" containerID="cri-o://7f707df0d0c7956f897746b384ed83c763c38678eeb884239a3700dced7b99e2" gracePeriod=30 Sep 09 12:24:28.340544 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.339371 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="offline" containerID="cri-o://b2801bc9846a229744a2339b4e3848fac865c9fdda4fe430859a3d3a64409ef3" gracePeriod=30 Sep 09 12:24:28.340544 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.339430 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="online" containerID="cri-o://f047ef4b46c2a9c922271c81fa184b1f5c68ccf897997de5fd4624f1e13c5e8c" gracePeriod=30 Sep 09 12:24:28.340544 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.339466 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="ui" containerID="cri-o://74dd162114b5e8026d8e7506bf33698440eaa232c7462e084408ca19e4611c3c" gracePeriod=30 Sep 09 12:24:28.743649 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.743612 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl"] Sep 09 12:24:28.747629 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.747608 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.750636 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.750608 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-tj97x\"" Sep 09 12:24:28.758924 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.758899 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl"] Sep 09 12:24:28.784287 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.784242 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.784453 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.784321 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.784453 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.784383 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be9acf77-05f7-4911-b5b2-bcfed0825698-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.784527 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.784492 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.784574 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.784529 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.784574 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.784548 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpj5n\" (UniqueName: \"kubernetes.io/projected/be9acf77-05f7-4911-b5b2-bcfed0825698-kube-api-access-wpj5n\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.885552 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.885518 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.885552 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.885558 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.885827 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.885578 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpj5n\" (UniqueName: \"kubernetes.io/projected/be9acf77-05f7-4911-b5b2-bcfed0825698-kube-api-access-wpj5n\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.885827 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.885621 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.885827 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.885664 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.885827 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:24:28.885671 2556 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 12:24:28.885827 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.885717 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be9acf77-05f7-4911-b5b2-bcfed0825698-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.885827 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:24:28.885745 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-ui-tls podName:be9acf77-05f7-4911-b5b2-bcfed0825698 nodeName:}" failed. No retries permitted until 2026-09-09 12:24:29.385722641 +0000 UTC m=+1325.065628811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-t98nl" (UID: "be9acf77-05f7-4911-b5b2-bcfed0825698") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 12:24:28.886164 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:24:28.885954 2556 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 12:24:28.886164 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:24:28.885979 2556 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 09 12:24:28.886164 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:24:28.886002 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-offline-tls podName:be9acf77-05f7-4911-b5b2-bcfed0825698 nodeName:}" failed. No retries permitted until 2026-09-09 12:24:29.38598619 +0000 UTC m=+1325.065892362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-t98nl" (UID: "be9acf77-05f7-4911-b5b2-bcfed0825698") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 12:24:28.886164 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:24:28.886029 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-registry-tls podName:be9acf77-05f7-4911-b5b2-bcfed0825698 nodeName:}" failed. No retries permitted until 2026-09-09 12:24:29.386014156 +0000 UTC m=+1325.065920335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-t98nl" (UID: "be9acf77-05f7-4911-b5b2-bcfed0825698") : secret "feast-simple-feast-setup-registry-tls" not found Sep 09 12:24:28.886164 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.886085 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be9acf77-05f7-4911-b5b2-bcfed0825698-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.888290 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.888260 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:28.894787 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:28.894763 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpj5n\" (UniqueName: \"kubernetes.io/projected/be9acf77-05f7-4911-b5b2-bcfed0825698-kube-api-access-wpj5n\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:29.390271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.390152 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:29.390271 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.390224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:29.390755 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.390304 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:29.392682 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.392657 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:29.392785 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.392677 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:29.392785 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.392677 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-t98nl\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:29.511112 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.511079 2556 generic.go:356] "Generic (PLEG): container finished" podID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerID="74dd162114b5e8026d8e7506bf33698440eaa232c7462e084408ca19e4611c3c" exitCode=0 Sep 09 12:24:29.511112 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.511104 2556 generic.go:356] "Generic (PLEG): container finished" podID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerID="f047ef4b46c2a9c922271c81fa184b1f5c68ccf897997de5fd4624f1e13c5e8c" exitCode=0 Sep 09 12:24:29.511329 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.511153 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerDied","Data":"74dd162114b5e8026d8e7506bf33698440eaa232c7462e084408ca19e4611c3c"} Sep 09 12:24:29.511329 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.511195 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerDied","Data":"f047ef4b46c2a9c922271c81fa184b1f5c68ccf897997de5fd4624f1e13c5e8c"} Sep 09 12:24:29.657744 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.657643 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:29.934108 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:29.934082 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl"] Sep 09 12:24:29.936576 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:24:29.936542 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe9acf77_05f7_4911_b5b2_bcfed0825698.slice/crio-8ab29c3a4dd29ec2392e47b29349700fe861c3aa3684a3077699b7e3fff4ba7c WatchSource:0}: Error finding container 8ab29c3a4dd29ec2392e47b29349700fe861c3aa3684a3077699b7e3fff4ba7c: Status 404 returned error can't find the container with id 8ab29c3a4dd29ec2392e47b29349700fe861c3aa3684a3077699b7e3fff4ba7c Sep 09 12:24:30.520090 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:30.520056 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerStarted","Data":"1e626a98302392add9992f46d0b4bbe2cd142bbeaaf8463784205b46ac80fad5"} Sep 09 12:24:30.520480 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:30.520099 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerStarted","Data":"8ab29c3a4dd29ec2392e47b29349700fe861c3aa3684a3077699b7e3fff4ba7c"} Sep 09 12:24:33.240973 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:33.240933 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 09 12:24:35.543221 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:35.543167 2556 generic.go:356] "Generic (PLEG): container finished" podID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerID="1e626a98302392add9992f46d0b4bbe2cd142bbeaaf8463784205b46ac80fad5" exitCode=0 Sep 09 12:24:35.543567 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:35.543240 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerDied","Data":"1e626a98302392add9992f46d0b4bbe2cd142bbeaaf8463784205b46ac80fad5"} Sep 09 12:24:36.253632 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:36.253587 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Sep 09 12:24:36.549426 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:36.549347 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerStarted","Data":"1b502d02ada0e288258f73f4e21a6b2e2c5b5a233419afe8b686d3a12e0c779f"} Sep 09 12:24:41.572708 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:41.572622 2556 generic.go:356] "Generic (PLEG): container finished" podID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerID="1b502d02ada0e288258f73f4e21a6b2e2c5b5a233419afe8b686d3a12e0c779f" exitCode=0 Sep 09 12:24:41.572708 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:41.572697 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerDied","Data":"1b502d02ada0e288258f73f4e21a6b2e2c5b5a233419afe8b686d3a12e0c779f"} Sep 09 12:24:42.583784 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:42.583747 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerStarted","Data":"5e05365e8ebfabcb7d6176edaec63c719aeb9595ba38ff376a9ebdea94f244fc"} Sep 09 12:24:42.584284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:42.583807 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerStarted","Data":"0d6a4752288f7c940144c70c429a8a299f223ff0310a16e49c85f7468bc8db14"} Sep 09 12:24:42.584284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:42.583829 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerStarted","Data":"40ea68f3c9ac4e95127dfeadd0a5a5fd2a4500a8847aee5dd22afcd8520745b8"} Sep 09 12:24:42.584284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:42.583847 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerStarted","Data":"261ff791741ce3349dfbc63fd89ec8ce6e67a375552e66ece8e89cc731f551e7"} Sep 09 12:24:42.611369 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:42.611305 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podStartSLOduration=14.611268675 podStartE2EDuration="14.611268675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:24:42.608505725 +0000 UTC m=+1338.288411914" watchObservedRunningTime="2026-09-09 12:24:42.611268675 +0000 UTC m=+1338.291174864" Sep 09 12:24:43.242061 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:43.242005 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 09 12:24:44.658713 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:44.658671 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:44.659172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:44.658738 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:44.659172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:44.658758 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:44.659172 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:44.658775 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:44.660901 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:44.660864 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 09 12:24:44.661052 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:44.660963 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused" Sep 09 12:24:44.661121 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:44.661058 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 12:24:44.661121 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:44.661112 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 09 12:24:46.253783 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:46.253728 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Sep 09 12:24:47.658450 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:47.658343 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 09 12:24:47.658914 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:47.658343 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 09 12:24:47.658914 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:47.658365 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused" Sep 09 12:24:47.658914 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:47.658682 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 12:24:49.658782 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:49.658730 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:49.659292 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:49.658800 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:49.659292 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:49.658820 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:49.659292 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:49.658835 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:50.658648 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:50.658611 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:50.659184 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:50.659162 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:50.659551 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:50.659301 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:50.659551 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:50.659479 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:50.664026 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:50.664010 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:51.638361 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:51.638330 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:51.638564 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:51.638506 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:51.640950 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:51.640930 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:24:53.241489 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:53.241447 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 09 12:24:53.241957 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:53.241570 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:24:56.253440 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:56.253397 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Sep 09 12:24:56.253883 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:56.253553 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:24:58.669670 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:58.669575 2556 generic.go:356] "Generic (PLEG): container finished" podID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerID="b2801bc9846a229744a2339b4e3848fac865c9fdda4fe430859a3d3a64409ef3" exitCode=137 Sep 09 12:24:58.669670 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:58.669607 2556 generic.go:356] "Generic (PLEG): container finished" podID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerID="7f707df0d0c7956f897746b384ed83c763c38678eeb884239a3700dced7b99e2" exitCode=137 Sep 09 12:24:58.669670 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:58.669615 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerDied","Data":"b2801bc9846a229744a2339b4e3848fac865c9fdda4fe430859a3d3a64409ef3"} Sep 09 12:24:58.669670 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:58.669663 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerDied","Data":"7f707df0d0c7956f897746b384ed83c763c38678eeb884239a3700dced7b99e2"} Sep 09 12:24:58.984161 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:58.984142 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:24:59.073746 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.073699 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-registry-tls\") pod \"adddf804-fcf8-41e6-bac0-1bf1d779837e\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " Sep 09 12:24:59.073900 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.073765 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-online-tls\") pod \"adddf804-fcf8-41e6-bac0-1bf1d779837e\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " Sep 09 12:24:59.073900 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.073875 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/adddf804-fcf8-41e6-bac0-1bf1d779837e-feast-data\") pod \"adddf804-fcf8-41e6-bac0-1bf1d779837e\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " Sep 09 12:24:59.073999 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.073912 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqw5w\" (UniqueName: \"kubernetes.io/projected/adddf804-fcf8-41e6-bac0-1bf1d779837e-kube-api-access-sqw5w\") pod \"adddf804-fcf8-41e6-bac0-1bf1d779837e\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " Sep 09 12:24:59.073999 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.073951 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-ui-tls\") pod \"adddf804-fcf8-41e6-bac0-1bf1d779837e\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " Sep 09 12:24:59.074113 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.074004 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-offline-tls\") pod \"adddf804-fcf8-41e6-bac0-1bf1d779837e\" (UID: \"adddf804-fcf8-41e6-bac0-1bf1d779837e\") " Sep 09 12:24:59.074502 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.074471 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adddf804-fcf8-41e6-bac0-1bf1d779837e-feast-data" (OuterVolumeSpecName: "feast-data") pod "adddf804-fcf8-41e6-bac0-1bf1d779837e" (UID: "adddf804-fcf8-41e6-bac0-1bf1d779837e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:24:59.076142 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.076110 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adddf804-fcf8-41e6-bac0-1bf1d779837e-kube-api-access-sqw5w" (OuterVolumeSpecName: "kube-api-access-sqw5w") pod "adddf804-fcf8-41e6-bac0-1bf1d779837e" (UID: "adddf804-fcf8-41e6-bac0-1bf1d779837e"). InnerVolumeSpecName "kube-api-access-sqw5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:24:59.076337 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.076311 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-online-tls" (OuterVolumeSpecName: "online-tls") pod "adddf804-fcf8-41e6-bac0-1bf1d779837e" (UID: "adddf804-fcf8-41e6-bac0-1bf1d779837e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:24:59.076414 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.076356 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "adddf804-fcf8-41e6-bac0-1bf1d779837e" (UID: "adddf804-fcf8-41e6-bac0-1bf1d779837e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:24:59.076486 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.076467 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "adddf804-fcf8-41e6-bac0-1bf1d779837e" (UID: "adddf804-fcf8-41e6-bac0-1bf1d779837e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:24:59.076579 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.076565 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "adddf804-fcf8-41e6-bac0-1bf1d779837e" (UID: "adddf804-fcf8-41e6-bac0-1bf1d779837e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:24:59.175441 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.175399 2556 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-online-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:24:59.175441 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.175439 2556 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/adddf804-fcf8-41e6-bac0-1bf1d779837e-feast-data\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:24:59.175441 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.175454 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqw5w\" (UniqueName: \"kubernetes.io/projected/adddf804-fcf8-41e6-bac0-1bf1d779837e-kube-api-access-sqw5w\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:24:59.175733 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.175464 2556 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-ui-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:24:59.175733 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.175474 2556 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-offline-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:24:59.175733 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.175482 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/adddf804-fcf8-41e6-bac0-1bf1d779837e-registry-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:24:59.676420 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.676394 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" Sep 09 12:24:59.676420 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.676400 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj" event={"ID":"adddf804-fcf8-41e6-bac0-1bf1d779837e","Type":"ContainerDied","Data":"15ea6a92b7f4a2c0ff85c2cffd6832e361318a8c94e938f25f5824e4e159b372"} Sep 09 12:24:59.676974 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.676449 2556 scope.go:117] "RemoveContainer" containerID="74dd162114b5e8026d8e7506bf33698440eaa232c7462e084408ca19e4611c3c" Sep 09 12:24:59.685712 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.685619 2556 scope.go:117] "RemoveContainer" containerID="b2801bc9846a229744a2339b4e3848fac865c9fdda4fe430859a3d3a64409ef3" Sep 09 12:24:59.694193 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.694173 2556 scope.go:117] "RemoveContainer" containerID="f047ef4b46c2a9c922271c81fa184b1f5c68ccf897997de5fd4624f1e13c5e8c" Sep 09 12:24:59.698660 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.698638 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj"] Sep 09 12:24:59.702037 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.702010 2556 scope.go:117] "RemoveContainer" containerID="7f707df0d0c7956f897746b384ed83c763c38678eeb884239a3700dced7b99e2" Sep 09 12:24:59.704776 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.704756 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qhnpj"] Sep 09 12:24:59.709566 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.709548 2556 scope.go:117] "RemoveContainer" containerID="846974733d631b7185f9b04ff050a782e7af34818e6f137ab7ff390831c05159" Sep 09 12:24:59.716527 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:24:59.716505 2556 scope.go:117] "RemoveContainer" containerID="732c11d8f89bdfd1cc2a704ea968be84fcd1e6aeb0b4a9ceb0b74b12f06cedf2" Sep 09 12:25:00.821585 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:00.821552 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" path="/var/lib/kubelet/pods/adddf804-fcf8-41e6-bac0-1bf1d779837e/volumes" Sep 09 12:25:53.694079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694035 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb"] Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694395 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="online" Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694409 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="online" Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694420 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="registry" Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694425 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="registry" Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694434 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="ui" Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694440 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="ui" Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694445 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="feast-apply" Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694451 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="feast-apply" Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694456 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="offline" Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694463 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="offline" Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694469 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="feast-init" Sep 09 12:25:53.694511 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694475 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="feast-init" Sep 09 12:25:53.694872 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694525 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="ui" Sep 09 12:25:53.694872 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694534 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="offline" Sep 09 12:25:53.694872 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694541 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="online" Sep 09 12:25:53.694872 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.694548 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="adddf804-fcf8-41e6-bac0-1bf1d779837e" containerName="registry" Sep 09 12:25:53.698691 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.698671 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.702118 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.702093 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 12:25:53.702262 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.702127 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 12:25:53.702262 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.702142 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 12:25:53.702262 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.702190 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 12:25:53.702262 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.702144 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-qwncc\"" Sep 09 12:25:53.702262 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.702103 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 12:25:53.702262 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.702099 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 12:25:53.723574 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.723555 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb"] Sep 09 12:25:53.736284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.736260 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmb8\" (UniqueName: \"kubernetes.io/projected/11f764cb-bbac-4d89-8a85-05bd24abcc33-kube-api-access-tbmb8\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.736374 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.736291 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/11f764cb-bbac-4d89-8a85-05bd24abcc33-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.736374 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.736314 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.736374 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.736348 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.736479 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.736422 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.736479 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.736458 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/11f764cb-bbac-4d89-8a85-05bd24abcc33-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.837003 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.836974 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmb8\" (UniqueName: \"kubernetes.io/projected/11f764cb-bbac-4d89-8a85-05bd24abcc33-kube-api-access-tbmb8\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.837003 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.837007 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/11f764cb-bbac-4d89-8a85-05bd24abcc33-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.837237 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.837030 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.837237 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.837045 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.837237 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.837073 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.837237 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.837100 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/11f764cb-bbac-4d89-8a85-05bd24abcc33-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.837237 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:25:53.837147 2556 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 09 12:25:53.837237 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:25:53.837153 2556 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 12:25:53.837501 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:25:53.837242 2556 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 12:25:53.837501 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:25:53.837244 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-offline-tls podName:11f764cb-bbac-4d89-8a85-05bd24abcc33 nodeName:}" failed. No retries permitted until 2026-09-09 12:25:54.337198877 +0000 UTC m=+1410.017105057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-offline-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-qjspb" (UID: "11f764cb-bbac-4d89-8a85-05bd24abcc33") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 09 12:25:53.837501 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:25:53.837299 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-ui-tls podName:11f764cb-bbac-4d89-8a85-05bd24abcc33 nodeName:}" failed. No retries permitted until 2026-09-09 12:25:54.337282593 +0000 UTC m=+1410.017188773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-qjspb" (UID: "11f764cb-bbac-4d89-8a85-05bd24abcc33") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 12:25:53.837501 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:25:53.837320 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-online-tls podName:11f764cb-bbac-4d89-8a85-05bd24abcc33 nodeName:}" failed. No retries permitted until 2026-09-09 12:25:54.337311464 +0000 UTC m=+1410.017217630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-online-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-qjspb" (UID: "11f764cb-bbac-4d89-8a85-05bd24abcc33") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 12:25:53.837644 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.837517 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/11f764cb-bbac-4d89-8a85-05bd24abcc33-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.837644 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.837637 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/11f764cb-bbac-4d89-8a85-05bd24abcc33-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:53.851379 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:53.851354 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmb8\" (UniqueName: \"kubernetes.io/projected/11f764cb-bbac-4d89-8a85-05bd24abcc33-kube-api-access-tbmb8\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:54.342039 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:54.342003 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:54.342039 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:54.342038 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:54.342283 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:54.342064 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:54.344423 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:54.344391 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:54.344541 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:54.344440 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:54.344541 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:54.344449 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-qjspb\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:54.607828 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:54.607734 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:25:54.885887 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:54.885863 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb"] Sep 09 12:25:54.886592 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:25:54.886564 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11f764cb_bbac_4d89_8a85_05bd24abcc33.slice/crio-3c5f72a6212ca2172965fa65e4d263ea5de998919bb4eba159af93803adf10a7 WatchSource:0}: Error finding container 3c5f72a6212ca2172965fa65e4d263ea5de998919bb4eba159af93803adf10a7: Status 404 returned error can't find the container with id 3c5f72a6212ca2172965fa65e4d263ea5de998919bb4eba159af93803adf10a7 Sep 09 12:25:54.900013 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:54.899989 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerStarted","Data":"3c5f72a6212ca2172965fa65e4d263ea5de998919bb4eba159af93803adf10a7"} Sep 09 12:25:55.905346 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:55.905307 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerStarted","Data":"0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d"} Sep 09 12:25:59.923029 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:59.922993 2556 generic.go:356] "Generic (PLEG): container finished" podID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerID="0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d" exitCode=0 Sep 09 12:25:59.923392 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:25:59.923067 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerDied","Data":"0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d"} Sep 09 12:26:00.929550 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:00.929517 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerStarted","Data":"c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967"} Sep 09 12:26:22.020804 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:22.020771 2556 generic.go:356] "Generic (PLEG): container finished" podID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerID="c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967" exitCode=0 Sep 09 12:26:22.021229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:22.020841 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerDied","Data":"c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967"} Sep 09 12:26:23.029932 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:23.029897 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerStarted","Data":"f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b"} Sep 09 12:26:23.030552 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:23.029950 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerStarted","Data":"b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168"} Sep 09 12:26:23.030552 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:23.029965 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerStarted","Data":"49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726"} Sep 09 12:26:23.054215 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:23.054140 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podStartSLOduration=30.05410738 podStartE2EDuration="30.05410738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 12:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:26:23.052479835 +0000 UTC m=+1438.732386023" watchObservedRunningTime="2026-09-09 12:26:23.05410738 +0000 UTC m=+1438.734013567" Sep 09 12:26:24.608917 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:24.608866 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:24.609431 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:24.609025 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:24.609431 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:24.609050 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:24.609431 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:24.609063 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:24.609431 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:24.609073 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:24.609431 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:24.609089 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:24.611087 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:24.611053 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 09 12:26:24.611234 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:24.611118 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 12:26:24.611234 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:24.611053 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused" Sep 09 12:26:27.608788 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:27.608742 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused" Sep 09 12:26:27.609317 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:27.608741 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 12:26:27.609317 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:27.608736 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 09 12:26:30.609598 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:30.609566 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:30.610132 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:30.610007 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:30.610325 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:30.610306 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:31.071407 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:31.071378 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:31.609419 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:31.609374 2556 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 09 12:26:34.317418 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:34.317388 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:35.002635 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:35.002605 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:26:35.612367 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:35.612329 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 12:26:55.612360 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:26:55.612314 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 12:27:15.613359 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:15.613306 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 12:27:35.612369 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:35.612317 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 12:27:35.819242 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:35.819187 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb"] Sep 09 12:27:35.819499 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:35.819445 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" containerID="cri-o://49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726" gracePeriod=30 Sep 09 12:27:35.819654 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:35.819502 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="offline" containerID="cri-o://b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168" gracePeriod=30 Sep 09 12:27:35.819654 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:35.819502 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="ui" containerID="cri-o://f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b" gracePeriod=30 Sep 09 12:27:36.012999 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:36.012967 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl"] Sep 09 12:27:36.013540 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:36.013275 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="registry" containerID="cri-o://261ff791741ce3349dfbc63fd89ec8ce6e67a375552e66ece8e89cc731f551e7" gracePeriod=30 Sep 09 12:27:36.013540 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:36.013343 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="offline" containerID="cri-o://0d6a4752288f7c940144c70c429a8a299f223ff0310a16e49c85f7468bc8db14" gracePeriod=30 Sep 09 12:27:36.013540 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:36.013384 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="online" containerID="cri-o://40ea68f3c9ac4e95127dfeadd0a5a5fd2a4500a8847aee5dd22afcd8520745b8" gracePeriod=30 Sep 09 12:27:36.013540 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:36.013482 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="ui" containerID="cri-o://5e05365e8ebfabcb7d6176edaec63c719aeb9595ba38ff376a9ebdea94f244fc" gracePeriod=30 Sep 09 12:27:37.345246 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:37.345191 2556 generic.go:356] "Generic (PLEG): container finished" podID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerID="5e05365e8ebfabcb7d6176edaec63c719aeb9595ba38ff376a9ebdea94f244fc" exitCode=0 Sep 09 12:27:37.345246 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:37.345230 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerDied","Data":"5e05365e8ebfabcb7d6176edaec63c719aeb9595ba38ff376a9ebdea94f244fc"} Sep 09 12:27:37.345705 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:37.345274 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerDied","Data":"40ea68f3c9ac4e95127dfeadd0a5a5fd2a4500a8847aee5dd22afcd8520745b8"} Sep 09 12:27:37.345705 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:37.345239 2556 generic.go:356] "Generic (PLEG): container finished" podID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerID="40ea68f3c9ac4e95127dfeadd0a5a5fd2a4500a8847aee5dd22afcd8520745b8" exitCode=0 Sep 09 12:27:37.347974 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:37.347949 2556 generic.go:356] "Generic (PLEG): container finished" podID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerID="f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b" exitCode=0 Sep 09 12:27:37.347974 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:37.347971 2556 generic.go:356] "Generic (PLEG): container finished" podID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerID="49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726" exitCode=0 Sep 09 12:27:37.348133 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:37.348007 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerDied","Data":"f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b"} Sep 09 12:27:37.348133 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:37.348033 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerDied","Data":"49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726"} Sep 09 12:27:40.610757 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:40.610708 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 09 12:27:41.638300 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:41.638253 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 09 12:27:41.638300 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:41.638290 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 12:27:44.318863 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:44.318821 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 12:27:50.610286 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:50.610245 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 09 12:27:51.638159 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:51.638114 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 09 12:27:51.638159 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:51.638148 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 12:27:54.318105 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:27:54.318051 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 12:28:00.610501 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:00.610459 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Sep 09 12:28:00.610903 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:00.610585 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:28:01.638801 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:01.638750 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Sep 09 12:28:01.639294 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:01.638767 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 12:28:01.639294 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:01.638885 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:28:01.639294 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:01.639054 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:28:04.318563 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:04.318523 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 12:28:04.318945 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:04.318629 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:28:05.999915 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:05.999890 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:28:06.106160 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.106106 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbmb8\" (UniqueName: \"kubernetes.io/projected/11f764cb-bbac-4d89-8a85-05bd24abcc33-kube-api-access-tbmb8\") pod \"11f764cb-bbac-4d89-8a85-05bd24abcc33\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " Sep 09 12:28:06.106160 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.106148 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-offline-tls\") pod \"11f764cb-bbac-4d89-8a85-05bd24abcc33\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " Sep 09 12:28:06.106349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.106177 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-online-tls\") pod \"11f764cb-bbac-4d89-8a85-05bd24abcc33\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " Sep 09 12:28:06.106349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.106222 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-ui-tls\") pod \"11f764cb-bbac-4d89-8a85-05bd24abcc33\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " Sep 09 12:28:06.106349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.106240 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/11f764cb-bbac-4d89-8a85-05bd24abcc33-feast-data\") pod \"11f764cb-bbac-4d89-8a85-05bd24abcc33\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " Sep 09 12:28:06.106349 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.106270 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/11f764cb-bbac-4d89-8a85-05bd24abcc33-registry-tls\") pod \"11f764cb-bbac-4d89-8a85-05bd24abcc33\" (UID: \"11f764cb-bbac-4d89-8a85-05bd24abcc33\") " Sep 09 12:28:06.106724 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.106699 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f764cb-bbac-4d89-8a85-05bd24abcc33-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "11f764cb-bbac-4d89-8a85-05bd24abcc33" (UID: "11f764cb-bbac-4d89-8a85-05bd24abcc33"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 12:28:06.106795 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.106766 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f764cb-bbac-4d89-8a85-05bd24abcc33-feast-data" (OuterVolumeSpecName: "feast-data") pod "11f764cb-bbac-4d89-8a85-05bd24abcc33" (UID: "11f764cb-bbac-4d89-8a85-05bd24abcc33"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:28:06.108346 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.108313 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-online-tls" (OuterVolumeSpecName: "online-tls") pod "11f764cb-bbac-4d89-8a85-05bd24abcc33" (UID: "11f764cb-bbac-4d89-8a85-05bd24abcc33"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:28:06.108436 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.108344 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "11f764cb-bbac-4d89-8a85-05bd24abcc33" (UID: "11f764cb-bbac-4d89-8a85-05bd24abcc33"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:28:06.108579 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.108565 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f764cb-bbac-4d89-8a85-05bd24abcc33-kube-api-access-tbmb8" (OuterVolumeSpecName: "kube-api-access-tbmb8") pod "11f764cb-bbac-4d89-8a85-05bd24abcc33" (UID: "11f764cb-bbac-4d89-8a85-05bd24abcc33"). InnerVolumeSpecName "kube-api-access-tbmb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:28:06.108622 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.108577 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "11f764cb-bbac-4d89-8a85-05bd24abcc33" (UID: "11f764cb-bbac-4d89-8a85-05bd24abcc33"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:28:06.206907 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.206875 2556 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-ui-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.206907 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.206902 2556 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/11f764cb-bbac-4d89-8a85-05bd24abcc33-feast-data\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.207081 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.206916 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/11f764cb-bbac-4d89-8a85-05bd24abcc33-registry-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.207081 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.206929 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmb8\" (UniqueName: \"kubernetes.io/projected/11f764cb-bbac-4d89-8a85-05bd24abcc33-kube-api-access-tbmb8\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.207081 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.206941 2556 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-offline-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.207081 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.206952 2556 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/11f764cb-bbac-4d89-8a85-05bd24abcc33-online-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.472996 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.472965 2556 generic.go:356] "Generic (PLEG): container finished" podID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerID="b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168" exitCode=137 Sep 09 12:28:06.473166 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.473008 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerDied","Data":"b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168"} Sep 09 12:28:06.473166 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.473028 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" Sep 09 12:28:06.473166 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.473065 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb" event={"ID":"11f764cb-bbac-4d89-8a85-05bd24abcc33","Type":"ContainerDied","Data":"3c5f72a6212ca2172965fa65e4d263ea5de998919bb4eba159af93803adf10a7"} Sep 09 12:28:06.473166 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.473091 2556 scope.go:117] "RemoveContainer" containerID="f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b" Sep 09 12:28:06.476547 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.476524 2556 generic.go:356] "Generic (PLEG): container finished" podID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerID="0d6a4752288f7c940144c70c429a8a299f223ff0310a16e49c85f7468bc8db14" exitCode=137 Sep 09 12:28:06.476547 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.476542 2556 generic.go:356] "Generic (PLEG): container finished" podID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerID="261ff791741ce3349dfbc63fd89ec8ce6e67a375552e66ece8e89cc731f551e7" exitCode=137 Sep 09 12:28:06.476707 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.476581 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerDied","Data":"0d6a4752288f7c940144c70c429a8a299f223ff0310a16e49c85f7468bc8db14"} Sep 09 12:28:06.476707 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.476614 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerDied","Data":"261ff791741ce3349dfbc63fd89ec8ce6e67a375552e66ece8e89cc731f551e7"} Sep 09 12:28:06.481501 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.481484 2556 scope.go:117] "RemoveContainer" containerID="b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168" Sep 09 12:28:06.489139 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.489123 2556 scope.go:117] "RemoveContainer" containerID="49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726" Sep 09 12:28:06.496194 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.496177 2556 scope.go:117] "RemoveContainer" containerID="c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967" Sep 09 12:28:06.500160 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.500139 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb"] Sep 09 12:28:06.503961 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.503941 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-qjspb"] Sep 09 12:28:06.504006 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.503948 2556 scope.go:117] "RemoveContainer" containerID="0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d" Sep 09 12:28:06.511122 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.511099 2556 scope.go:117] "RemoveContainer" containerID="f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b" Sep 09 12:28:06.511464 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:06.511446 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b\": container with ID starting with f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b not found: ID does not exist" containerID="f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b" Sep 09 12:28:06.511534 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.511471 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b"} err="failed to get container status \"f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b\": rpc error: code = NotFound desc = could not find container \"f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b\": container with ID starting with f20de87286079bcf69f0a2f8cd6e1afaa9c74544e9f24b727eb8c8dc11e8078b not found: ID does not exist" Sep 09 12:28:06.511534 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.511489 2556 scope.go:117] "RemoveContainer" containerID="b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168" Sep 09 12:28:06.511722 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:06.511708 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168\": container with ID starting with b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168 not found: ID does not exist" containerID="b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168" Sep 09 12:28:06.511756 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.511725 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168"} err="failed to get container status \"b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168\": rpc error: code = NotFound desc = could not find container \"b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168\": container with ID starting with b8b26088d2f0a64de37ac7cbdf88741af11609a5ffb62dc3c98f32dce8b3d168 not found: ID does not exist" Sep 09 12:28:06.511756 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.511737 2556 scope.go:117] "RemoveContainer" containerID="49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726" Sep 09 12:28:06.511970 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:06.511957 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726\": container with ID starting with 49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726 not found: ID does not exist" containerID="49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726" Sep 09 12:28:06.512012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.511972 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726"} err="failed to get container status \"49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726\": rpc error: code = NotFound desc = could not find container \"49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726\": container with ID starting with 49973dac6ee10b02b05e07d080e86e3f18a5f0f1844c9265e6b7dfb2cbff7726 not found: ID does not exist" Sep 09 12:28:06.512012 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.511983 2556 scope.go:117] "RemoveContainer" containerID="c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967" Sep 09 12:28:06.512221 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:06.512186 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967\": container with ID starting with c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967 not found: ID does not exist" containerID="c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967" Sep 09 12:28:06.512263 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.512232 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967"} err="failed to get container status \"c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967\": rpc error: code = NotFound desc = could not find container \"c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967\": container with ID starting with c8a2e954248dc1c535a5f668fa245aebc24e641a545305f02653100c04bab967 not found: ID does not exist" Sep 09 12:28:06.512263 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.512245 2556 scope.go:117] "RemoveContainer" containerID="0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d" Sep 09 12:28:06.512481 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:06.512463 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d\": container with ID starting with 0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d not found: ID does not exist" containerID="0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d" Sep 09 12:28:06.512521 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.512488 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d"} err="failed to get container status \"0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d\": rpc error: code = NotFound desc = could not find container \"0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d\": container with ID starting with 0384035189e0aa5ee5f76531cf10daae34a9f0ef643b61b547a7c5f3b34e466d not found: ID does not exist" Sep 09 12:28:06.649302 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.649281 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:28:06.710384 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.710358 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpj5n\" (UniqueName: \"kubernetes.io/projected/be9acf77-05f7-4911-b5b2-bcfed0825698-kube-api-access-wpj5n\") pod \"be9acf77-05f7-4911-b5b2-bcfed0825698\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " Sep 09 12:28:06.710522 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.710402 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be9acf77-05f7-4911-b5b2-bcfed0825698-feast-data\") pod \"be9acf77-05f7-4911-b5b2-bcfed0825698\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " Sep 09 12:28:06.710522 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.710431 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-online-tls\") pod \"be9acf77-05f7-4911-b5b2-bcfed0825698\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " Sep 09 12:28:06.710522 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.710458 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-ui-tls\") pod \"be9acf77-05f7-4911-b5b2-bcfed0825698\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " Sep 09 12:28:06.710522 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.710475 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-offline-tls\") pod \"be9acf77-05f7-4911-b5b2-bcfed0825698\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " Sep 09 12:28:06.710522 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.710491 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-registry-tls\") pod \"be9acf77-05f7-4911-b5b2-bcfed0825698\" (UID: \"be9acf77-05f7-4911-b5b2-bcfed0825698\") " Sep 09 12:28:06.710952 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.710924 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9acf77-05f7-4911-b5b2-bcfed0825698-feast-data" (OuterVolumeSpecName: "feast-data") pod "be9acf77-05f7-4911-b5b2-bcfed0825698" (UID: "be9acf77-05f7-4911-b5b2-bcfed0825698"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:28:06.712565 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.712537 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-online-tls" (OuterVolumeSpecName: "online-tls") pod "be9acf77-05f7-4911-b5b2-bcfed0825698" (UID: "be9acf77-05f7-4911-b5b2-bcfed0825698"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:28:06.712693 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.712589 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "be9acf77-05f7-4911-b5b2-bcfed0825698" (UID: "be9acf77-05f7-4911-b5b2-bcfed0825698"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:28:06.712693 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.712619 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "be9acf77-05f7-4911-b5b2-bcfed0825698" (UID: "be9acf77-05f7-4911-b5b2-bcfed0825698"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:28:06.712837 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.712822 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9acf77-05f7-4911-b5b2-bcfed0825698-kube-api-access-wpj5n" (OuterVolumeSpecName: "kube-api-access-wpj5n") pod "be9acf77-05f7-4911-b5b2-bcfed0825698" (UID: "be9acf77-05f7-4911-b5b2-bcfed0825698"). InnerVolumeSpecName "kube-api-access-wpj5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:28:06.712924 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.712912 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "be9acf77-05f7-4911-b5b2-bcfed0825698" (UID: "be9acf77-05f7-4911-b5b2-bcfed0825698"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 12:28:06.811553 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.811493 2556 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be9acf77-05f7-4911-b5b2-bcfed0825698-feast-data\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.811553 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.811517 2556 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-online-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.811553 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.811527 2556 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-ui-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.811553 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.811536 2556 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-offline-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.811553 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.811544 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be9acf77-05f7-4911-b5b2-bcfed0825698-registry-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.811553 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.811551 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpj5n\" (UniqueName: \"kubernetes.io/projected/be9acf77-05f7-4911-b5b2-bcfed0825698-kube-api-access-wpj5n\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:06.816179 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:06.816155 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" path="/var/lib/kubelet/pods/11f764cb-bbac-4d89-8a85-05bd24abcc33/volumes" Sep 09 12:28:07.483026 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:07.482996 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" Sep 09 12:28:07.483026 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:07.483008 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl" event={"ID":"be9acf77-05f7-4911-b5b2-bcfed0825698","Type":"ContainerDied","Data":"8ab29c3a4dd29ec2392e47b29349700fe861c3aa3684a3077699b7e3fff4ba7c"} Sep 09 12:28:07.483572 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:07.483061 2556 scope.go:117] "RemoveContainer" containerID="5e05365e8ebfabcb7d6176edaec63c719aeb9595ba38ff376a9ebdea94f244fc" Sep 09 12:28:07.491266 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:07.491249 2556 scope.go:117] "RemoveContainer" containerID="0d6a4752288f7c940144c70c429a8a299f223ff0310a16e49c85f7468bc8db14" Sep 09 12:28:07.498802 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:07.498786 2556 scope.go:117] "RemoveContainer" containerID="40ea68f3c9ac4e95127dfeadd0a5a5fd2a4500a8847aee5dd22afcd8520745b8" Sep 09 12:28:07.502611 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:07.502591 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl"] Sep 09 12:28:07.506875 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:07.506850 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-t98nl"] Sep 09 12:28:07.507316 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:07.507294 2556 scope.go:117] "RemoveContainer" containerID="261ff791741ce3349dfbc63fd89ec8ce6e67a375552e66ece8e89cc731f551e7" Sep 09 12:28:07.514466 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:07.514450 2556 scope.go:117] "RemoveContainer" containerID="1b502d02ada0e288258f73f4e21a6b2e2c5b5a233419afe8b686d3a12e0c779f" Sep 09 12:28:07.521327 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:07.521309 2556 scope.go:117] "RemoveContainer" containerID="1e626a98302392add9992f46d0b4bbe2cd142bbeaaf8463784205b46ac80fad5" Sep 09 12:28:08.816155 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:08.816116 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" path="/var/lib/kubelet/pods/be9acf77-05f7-4911-b5b2-bcfed0825698/volumes" Sep 09 12:28:21.504702 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.504665 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vv65/must-gather-vrvcm"] Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.504976 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="feast-apply" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.504989 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="feast-apply" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.504999 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505005 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505013 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="offline" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505019 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="offline" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505029 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="feast-init" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505034 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="feast-init" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505039 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="ui" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505044 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="ui" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505050 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="feast-init" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505055 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="feast-init" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505062 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="online" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505067 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="online" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505073 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="ui" Sep 09 12:28:21.505079 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505078 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="ui" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505088 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="feast-apply" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505093 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="feast-apply" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505100 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="offline" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505105 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="offline" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505115 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="registry" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505119 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="registry" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505176 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="online" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505183 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="registry" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505188 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="offline" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505194 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f764cb-bbac-4d89-8a85-05bd24abcc33" containerName="ui" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505217 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="offline" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505223 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="ui" Sep 09 12:28:21.505602 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.505230 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9acf77-05f7-4911-b5b2-bcfed0825698" containerName="online" Sep 09 12:28:21.509690 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.509670 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:21.512468 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.512447 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vv65\"/\"openshift-service-ca.crt\"" Sep 09 12:28:21.513532 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.513512 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2vv65\"/\"default-dockercfg-h6sg5\"" Sep 09 12:28:21.513532 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.513530 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vv65\"/\"kube-root-ca.crt\"" Sep 09 12:28:21.517983 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.517963 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vv65/must-gather-vrvcm"] Sep 09 12:28:21.641148 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.641112 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qcpg\" (UniqueName: \"kubernetes.io/projected/ba454f18-40af-4137-bc1a-d340d3a305e9-kube-api-access-2qcpg\") pod \"must-gather-vrvcm\" (UID: \"ba454f18-40af-4137-bc1a-d340d3a305e9\") " pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:21.641380 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.641155 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ba454f18-40af-4137-bc1a-d340d3a305e9-must-gather-output\") pod \"must-gather-vrvcm\" (UID: \"ba454f18-40af-4137-bc1a-d340d3a305e9\") " pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:21.742327 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.742287 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qcpg\" (UniqueName: \"kubernetes.io/projected/ba454f18-40af-4137-bc1a-d340d3a305e9-kube-api-access-2qcpg\") pod \"must-gather-vrvcm\" (UID: \"ba454f18-40af-4137-bc1a-d340d3a305e9\") " pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:21.742327 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.742327 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ba454f18-40af-4137-bc1a-d340d3a305e9-must-gather-output\") pod \"must-gather-vrvcm\" (UID: \"ba454f18-40af-4137-bc1a-d340d3a305e9\") " pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:21.742644 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.742628 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ba454f18-40af-4137-bc1a-d340d3a305e9-must-gather-output\") pod \"must-gather-vrvcm\" (UID: \"ba454f18-40af-4137-bc1a-d340d3a305e9\") " pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:21.751035 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.751009 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qcpg\" (UniqueName: \"kubernetes.io/projected/ba454f18-40af-4137-bc1a-d340d3a305e9-kube-api-access-2qcpg\") pod \"must-gather-vrvcm\" (UID: \"ba454f18-40af-4137-bc1a-d340d3a305e9\") " pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:21.834011 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:21.833940 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:22.111248 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:22.111222 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vv65/must-gather-vrvcm"] Sep 09 12:28:22.113034 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:28:22.113007 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba454f18_40af_4137_bc1a_d340d3a305e9.slice/crio-020d7265f3dba3bd88306ba06328fd34d4d9afeb833f150612c151112072fe87 WatchSource:0}: Error finding container 020d7265f3dba3bd88306ba06328fd34d4d9afeb833f150612c151112072fe87: Status 404 returned error can't find the container with id 020d7265f3dba3bd88306ba06328fd34d4d9afeb833f150612c151112072fe87 Sep 09 12:28:22.114763 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:22.114750 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 12:28:22.547293 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:22.547256 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vv65/must-gather-vrvcm" event={"ID":"ba454f18-40af-4137-bc1a-d340d3a305e9","Type":"ContainerStarted","Data":"020d7265f3dba3bd88306ba06328fd34d4d9afeb833f150612c151112072fe87"} Sep 09 12:28:28.580503 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:28.580462 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vv65/must-gather-vrvcm" event={"ID":"ba454f18-40af-4137-bc1a-d340d3a305e9","Type":"ContainerStarted","Data":"d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d"} Sep 09 12:28:28.580915 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:28.580517 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vv65/must-gather-vrvcm" event={"ID":"ba454f18-40af-4137-bc1a-d340d3a305e9","Type":"ContainerStarted","Data":"6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16"} Sep 09 12:28:38.624838 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:38.624802 2556 generic.go:356] "Generic (PLEG): container finished" podID="ba454f18-40af-4137-bc1a-d340d3a305e9" containerID="6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16" exitCode=0 Sep 09 12:28:38.625284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:38.624849 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vv65/must-gather-vrvcm" event={"ID":"ba454f18-40af-4137-bc1a-d340d3a305e9","Type":"ContainerDied","Data":"6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16"} Sep 09 12:28:38.625284 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:38.625126 2556 scope.go:117] "RemoveContainer" containerID="6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16" Sep 09 12:28:38.748435 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:38.748384 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2vv65_must-gather-vrvcm_ba454f18-40af-4137-bc1a-d340d3a305e9/gather/0.log" Sep 09 12:28:42.838305 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:42.838273 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kzvf5_556aa98a-d697-4e96-beda-0ae8d694b31d/global-pull-secret-syncer/0.log" Sep 09 12:28:43.015993 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:43.015955 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qzjgn_ad4e6005-53a2-457e-8d11-5fdf78c799b2/konnectivity-agent/0.log" Sep 09 12:28:43.123272 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:43.123176 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-139.ec2.internal_968730223ffcdc713c2b8b2489f9ba2f/haproxy/0.log" Sep 09 12:28:44.072146 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.072109 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-2vv65/must-gather-vrvcm"] Sep 09 12:28:44.072605 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.072303 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-2vv65/must-gather-vrvcm" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" containerName="copy" containerID="cri-o://d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d" gracePeriod=2 Sep 09 12:28:44.078990 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.078958 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-2vv65/must-gather-vrvcm"] Sep 09 12:28:44.079660 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:44.079632 2556 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ba454f18-40af-4137-bc1a-d340d3a305e9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-2vv65\"/\"must-gather-vrvcm\": pods \"must-gather-vrvcm\" not found" pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:44.303678 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.303658 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2vv65_must-gather-vrvcm_ba454f18-40af-4137-bc1a-d340d3a305e9/copy/0.log" Sep 09 12:28:44.305160 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.305145 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:44.307438 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:44.307415 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vrvcm\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2vv65\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:44.352864 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.352813 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ba454f18-40af-4137-bc1a-d340d3a305e9-must-gather-output\") pod \"ba454f18-40af-4137-bc1a-d340d3a305e9\" (UID: \"ba454f18-40af-4137-bc1a-d340d3a305e9\") " Sep 09 12:28:44.352943 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.352902 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qcpg\" (UniqueName: \"kubernetes.io/projected/ba454f18-40af-4137-bc1a-d340d3a305e9-kube-api-access-2qcpg\") pod \"ba454f18-40af-4137-bc1a-d340d3a305e9\" (UID: \"ba454f18-40af-4137-bc1a-d340d3a305e9\") " Sep 09 12:28:44.353159 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.353138 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba454f18-40af-4137-bc1a-d340d3a305e9-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "ba454f18-40af-4137-bc1a-d340d3a305e9" (UID: "ba454f18-40af-4137-bc1a-d340d3a305e9"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 12:28:44.354903 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.354876 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba454f18-40af-4137-bc1a-d340d3a305e9-kube-api-access-2qcpg" (OuterVolumeSpecName: "kube-api-access-2qcpg") pod "ba454f18-40af-4137-bc1a-d340d3a305e9" (UID: "ba454f18-40af-4137-bc1a-d340d3a305e9"). InnerVolumeSpecName "kube-api-access-2qcpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 12:28:44.453655 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.453621 2556 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ba454f18-40af-4137-bc1a-d340d3a305e9-must-gather-output\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:44.453655 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.453649 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qcpg\" (UniqueName: \"kubernetes.io/projected/ba454f18-40af-4137-bc1a-d340d3a305e9-kube-api-access-2qcpg\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\"" Sep 09 12:28:44.651342 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.651269 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2vv65_must-gather-vrvcm_ba454f18-40af-4137-bc1a-d340d3a305e9/copy/0.log" Sep 09 12:28:44.651639 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.651619 2556 generic.go:356] "Generic (PLEG): container finished" podID="ba454f18-40af-4137-bc1a-d340d3a305e9" containerID="d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d" exitCode=143 Sep 09 12:28:44.651691 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.651663 2556 scope.go:117] "RemoveContainer" containerID="d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d" Sep 09 12:28:44.651691 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.651664 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:44.654051 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:44.654030 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vrvcm\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2vv65\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:44.659506 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.659492 2556 scope.go:117] "RemoveContainer" containerID="6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16" Sep 09 12:28:44.661380 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:44.661351 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vrvcm\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2vv65\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:44.670918 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.670901 2556 scope.go:117] "RemoveContainer" containerID="d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d" Sep 09 12:28:44.671162 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:44.671144 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d\": container with ID starting with d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d not found: ID does not exist" containerID="d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d" Sep 09 12:28:44.671229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.671170 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d"} err="failed to get container status \"d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d\": rpc error: code = NotFound desc = could not find container \"d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d\": container with ID starting with d4b681eab26d1297d7471721febeda7f1d212969e5cd43fae919c4c8d676f28d not found: ID does not exist" Sep 09 12:28:44.671229 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.671188 2556 scope.go:117] "RemoveContainer" containerID="6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16" Sep 09 12:28:44.671423 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:44.671409 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16\": container with ID starting with 6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16 not found: ID does not exist" containerID="6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16" Sep 09 12:28:44.671467 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.671426 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16"} err="failed to get container status \"6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16\": rpc error: code = NotFound desc = could not find container \"6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16\": container with ID starting with 6c73bf9ba562f5161552e40e19d6ad5c7a1f679a0f4c577abada9b6f286e8d16 not found: ID does not exist" Sep 09 12:28:44.815089 ip-10-0-143-139 kubenswrapper[2556]: E0909 12:28:44.815056 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vrvcm\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2vv65\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" pod="openshift-must-gather-2vv65/must-gather-vrvcm" Sep 09 12:28:44.816046 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:44.816026 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" path="/var/lib/kubelet/pods/ba454f18-40af-4137-bc1a-d340d3a305e9/volumes" Sep 09 12:28:47.069114 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.069082 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98045cb3-41cc-4912-95d0-8b4c30178891/alertmanager/0.log" Sep 09 12:28:47.092595 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.092531 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98045cb3-41cc-4912-95d0-8b4c30178891/config-reloader/0.log" Sep 09 12:28:47.119921 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.119898 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98045cb3-41cc-4912-95d0-8b4c30178891/kube-rbac-proxy-web/0.log" Sep 09 12:28:47.145974 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.145953 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98045cb3-41cc-4912-95d0-8b4c30178891/kube-rbac-proxy/0.log" Sep 09 12:28:47.170174 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.170149 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98045cb3-41cc-4912-95d0-8b4c30178891/kube-rbac-proxy-metric/0.log" Sep 09 12:28:47.191134 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.191106 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98045cb3-41cc-4912-95d0-8b4c30178891/prom-label-proxy/0.log" Sep 09 12:28:47.213276 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.213257 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_98045cb3-41cc-4912-95d0-8b4c30178891/init-config-reloader/0.log" Sep 09 12:28:47.292576 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.292324 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-t7nkg_e41430b3-e44e-4e0c-8b90-ac215547e15f/kube-state-metrics/0.log" Sep 09 12:28:47.310768 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.310746 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-t7nkg_e41430b3-e44e-4e0c-8b90-ac215547e15f/kube-rbac-proxy-main/0.log" Sep 09 12:28:47.335604 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.335583 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-t7nkg_e41430b3-e44e-4e0c-8b90-ac215547e15f/kube-rbac-proxy-self/0.log" Sep 09 12:28:47.363399 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.363324 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b99569b85-vhd4m_0244901a-7f18-428d-b6a1-bcfc699f8d10/metrics-server/0.log" Sep 09 12:28:47.388144 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.388112 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-pt754_c7f68e41-8cfb-46c0-872d-c4edc20b9032/monitoring-plugin/0.log" Sep 09 12:28:47.419323 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.419298 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rxfq_52be1a8c-fc1d-40ab-8294-c75bae298228/node-exporter/0.log" Sep 09 12:28:47.438779 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.438756 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rxfq_52be1a8c-fc1d-40ab-8294-c75bae298228/kube-rbac-proxy/0.log" Sep 09 12:28:47.463979 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.463960 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rxfq_52be1a8c-fc1d-40ab-8294-c75bae298228/init-textfile/0.log" Sep 09 12:28:47.648821 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.648737 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gbngg_5d0a2e6f-20b8-44d8-98cf-9433a0604560/kube-rbac-proxy-main/0.log" Sep 09 12:28:47.671347 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.671322 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gbngg_5d0a2e6f-20b8-44d8-98cf-9433a0604560/kube-rbac-proxy-self/0.log" Sep 09 12:28:47.693866 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.693827 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gbngg_5d0a2e6f-20b8-44d8-98cf-9433a0604560/openshift-state-metrics/0.log" Sep 09 12:28:47.733410 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.733387 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c8402e2-f17b-4fb9-9903-3cef7e063b65/prometheus/0.log" Sep 09 12:28:47.754286 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.754262 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c8402e2-f17b-4fb9-9903-3cef7e063b65/config-reloader/0.log" Sep 09 12:28:47.775889 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.775870 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c8402e2-f17b-4fb9-9903-3cef7e063b65/thanos-sidecar/0.log" Sep 09 12:28:47.798904 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.798884 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c8402e2-f17b-4fb9-9903-3cef7e063b65/kube-rbac-proxy-web/0.log" Sep 09 12:28:47.820906 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.820888 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c8402e2-f17b-4fb9-9903-3cef7e063b65/kube-rbac-proxy/0.log" Sep 09 12:28:47.844793 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.844773 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c8402e2-f17b-4fb9-9903-3cef7e063b65/kube-rbac-proxy-thanos/0.log" Sep 09 12:28:47.866533 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.866515 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c8402e2-f17b-4fb9-9903-3cef7e063b65/init-config-reloader/0.log" Sep 09 12:28:47.988530 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:47.988492 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ffbd7df6-8r5pj_9ad49d01-13b7-4e87-abf6-650375a53c27/telemeter-client/0.log" Sep 09 12:28:48.009368 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:48.009346 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ffbd7df6-8r5pj_9ad49d01-13b7-4e87-abf6-650375a53c27/reload/0.log" Sep 09 12:28:48.038449 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:48.038422 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ffbd7df6-8r5pj_9ad49d01-13b7-4e87-abf6-650375a53c27/kube-rbac-proxy/0.log" Sep 09 12:28:48.077957 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:48.077935 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65bb57b978-szsdl_4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b/thanos-query/0.log" Sep 09 12:28:48.106055 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:48.106026 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65bb57b978-szsdl_4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b/kube-rbac-proxy-web/0.log" Sep 09 12:28:48.135286 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:48.135261 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65bb57b978-szsdl_4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b/kube-rbac-proxy/0.log" Sep 09 12:28:48.160157 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:48.160134 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65bb57b978-szsdl_4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b/prom-label-proxy/0.log" Sep 09 12:28:48.184507 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:48.184490 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65bb57b978-szsdl_4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b/kube-rbac-proxy-rules/0.log" Sep 09 12:28:48.221113 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:48.221092 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65bb57b978-szsdl_4a7cb4d6-ee19-4df1-9825-5e0846ff1c4b/kube-rbac-proxy-metrics/0.log" Sep 09 12:28:50.277045 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.277022 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-dbbdf8bf8-b6wwq_9a55c776-36da-4709-9c8b-71e32b41749e/console/0.log" Sep 09 12:28:50.459386 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.459356 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f"] Sep 09 12:28:50.459684 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.459672 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" containerName="copy" Sep 09 12:28:50.459729 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.459686 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" containerName="copy" Sep 09 12:28:50.459729 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.459702 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" containerName="gather" Sep 09 12:28:50.459729 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.459708 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" containerName="gather" Sep 09 12:28:50.459827 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.459757 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" containerName="copy" Sep 09 12:28:50.459827 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.459767 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba454f18-40af-4137-bc1a-d340d3a305e9" containerName="gather" Sep 09 12:28:50.463917 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.463898 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.470167 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.470148 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v7zj2\"/\"default-dockercfg-qjp6x\"" Sep 09 12:28:50.470972 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.470959 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v7zj2\"/\"openshift-service-ca.crt\"" Sep 09 12:28:50.476584 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.471805 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v7zj2\"/\"kube-root-ca.crt\"" Sep 09 12:28:50.481055 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.481033 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f"] Sep 09 12:28:50.499837 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.499812 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-sys\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.499837 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.499837 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-proc\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.499967 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.499875 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-podres\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.499967 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.499951 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-lib-modules\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.500038 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.499974 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h8gs\" (UniqueName: \"kubernetes.io/projected/31df5def-4dcf-4011-a510-d9d7abf22c92-kube-api-access-5h8gs\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.601355 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.601291 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-podres\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.601355 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.601332 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-lib-modules\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.601355 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.601349 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h8gs\" (UniqueName: \"kubernetes.io/projected/31df5def-4dcf-4011-a510-d9d7abf22c92-kube-api-access-5h8gs\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.601572 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.601388 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-sys\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.601572 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.601402 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-proc\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.601572 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.601471 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-proc\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.601572 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.601473 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-podres\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.601572 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.601473 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-lib-modules\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.601572 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.601507 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31df5def-4dcf-4011-a510-d9d7abf22c92-sys\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.621868 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.621842 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h8gs\" (UniqueName: \"kubernetes.io/projected/31df5def-4dcf-4011-a510-d9d7abf22c92-kube-api-access-5h8gs\") pod \"perf-node-gather-daemonset-2zl2f\" (UID: \"31df5def-4dcf-4011-a510-d9d7abf22c92\") " pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:50.777742 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:50.777711 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:51.046616 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:51.046557 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f"] Sep 09 12:28:51.048988 ip-10-0-143-139 kubenswrapper[2556]: W0909 12:28:51.048962 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod31df5def_4dcf_4011_a510_d9d7abf22c92.slice/crio-051ddf199aebb227a499cf7ab205331755747f04fcbee21f363c57dfd0654d13 WatchSource:0}: Error finding container 051ddf199aebb227a499cf7ab205331755747f04fcbee21f363c57dfd0654d13: Status 404 returned error can't find the container with id 051ddf199aebb227a499cf7ab205331755747f04fcbee21f363c57dfd0654d13 Sep 09 12:28:51.681187 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:51.681151 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" event={"ID":"31df5def-4dcf-4011-a510-d9d7abf22c92","Type":"ContainerStarted","Data":"af277788c29ae046eccac9883f0bab9596e138985f2d47418646f3bbfe1cd462"} Sep 09 12:28:51.681564 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:51.681225 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" event={"ID":"31df5def-4dcf-4011-a510-d9d7abf22c92","Type":"ContainerStarted","Data":"051ddf199aebb227a499cf7ab205331755747f04fcbee21f363c57dfd0654d13"} Sep 09 12:28:51.681564 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:51.681256 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:51.697826 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:51.697765 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" podStartSLOduration=1.6977504190000001 podStartE2EDuration="1.697750419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 12:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 12:28:51.696189872 +0000 UTC m=+1587.376096058" watchObservedRunningTime="2026-09-09 12:28:51.697750419 +0000 UTC m=+1587.377656606" Sep 09 12:28:51.809001 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:51.808978 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r564d_584516fc-2718-462e-8753-491fa88197d0/dns/0.log" Sep 09 12:28:51.831859 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:51.831838 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r564d_584516fc-2718-462e-8753-491fa88197d0/kube-rbac-proxy/0.log" Sep 09 12:28:51.876732 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:51.876704 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbf5s_0e051d05-2b45-4348-86a0-0e9e2a31514f/dns-node-resolver/0.log" Sep 09 12:28:52.342537 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:52.342513 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6dx5z_c6c86f8c-76fb-47ba-9189-e02f1bfb574b/node-ca/0.log" Sep 09 12:28:53.452198 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:53.452167 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9blz8_3a3d2c78-1cae-4271-8944-0437fd466a3b/serve-healthcheck-canary/0.log" Sep 09 12:28:53.913256 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:53.913166 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-96bsw_f65bb66e-9ad6-4ebb-9261-3688257fecaf/kube-rbac-proxy/0.log" Sep 09 12:28:53.931840 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:53.931817 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-96bsw_f65bb66e-9ad6-4ebb-9261-3688257fecaf/exporter/0.log" Sep 09 12:28:53.952283 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:53.952263 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-96bsw_f65bb66e-9ad6-4ebb-9261-3688257fecaf/extractor/0.log" Sep 09 12:28:57.694388 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:57.694361 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v7zj2/perf-node-gather-daemonset-2zl2f" Sep 09 12:28:59.639312 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:59.639285 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8cpgt_166bdd7f-1ca5-459d-adfa-a0f76bd75a57/kube-multus-additional-cni-plugins/0.log" Sep 09 12:28:59.660181 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:59.660155 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8cpgt_166bdd7f-1ca5-459d-adfa-a0f76bd75a57/egress-router-binary-copy/0.log" Sep 09 12:28:59.682693 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:59.682652 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8cpgt_166bdd7f-1ca5-459d-adfa-a0f76bd75a57/cni-plugins/0.log" Sep 09 12:28:59.705421 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:59.705392 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8cpgt_166bdd7f-1ca5-459d-adfa-a0f76bd75a57/bond-cni-plugin/0.log" Sep 09 12:28:59.725881 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:59.725856 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8cpgt_166bdd7f-1ca5-459d-adfa-a0f76bd75a57/routeoverride-cni/0.log" Sep 09 12:28:59.745372 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:59.745349 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8cpgt_166bdd7f-1ca5-459d-adfa-a0f76bd75a57/whereabouts-cni-bincopy/0.log" Sep 09 12:28:59.767917 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:59.767892 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8cpgt_166bdd7f-1ca5-459d-adfa-a0f76bd75a57/whereabouts-cni/0.log" Sep 09 12:28:59.996526 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:28:59.996498 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kgsmn_210607a5-ef5e-40d2-a314-03329bcb4064/kube-multus/0.log" Sep 09 12:29:00.175553 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:00.175525 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rktvs_40b400cd-b876-4e32-b6c1-a1322eb20817/network-metrics-daemon/0.log" Sep 09 12:29:00.193678 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:00.193580 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rktvs_40b400cd-b876-4e32-b6c1-a1322eb20817/kube-rbac-proxy/0.log" Sep 09 12:29:01.117416 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:01.117382 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncg4_a151e674-cd4a-460a-bc4d-06d31ce26adc/ovn-controller/0.log" Sep 09 12:29:01.134794 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:01.134765 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncg4_a151e674-cd4a-460a-bc4d-06d31ce26adc/ovn-acl-logging/0.log" Sep 09 12:29:01.156710 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:01.156683 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncg4_a151e674-cd4a-460a-bc4d-06d31ce26adc/kube-rbac-proxy-node/0.log" Sep 09 12:29:01.176222 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:01.176180 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncg4_a151e674-cd4a-460a-bc4d-06d31ce26adc/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 12:29:01.194181 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:01.194161 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncg4_a151e674-cd4a-460a-bc4d-06d31ce26adc/northd/0.log" Sep 09 12:29:01.213360 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:01.213336 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncg4_a151e674-cd4a-460a-bc4d-06d31ce26adc/nbdb/0.log" Sep 09 12:29:01.231879 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:01.231855 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncg4_a151e674-cd4a-460a-bc4d-06d31ce26adc/sbdb/0.log" Sep 09 12:29:01.285815 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:01.285789 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lncg4_a151e674-cd4a-460a-bc4d-06d31ce26adc/ovnkube-controller/0.log" Sep 09 12:29:02.448551 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:02.448521 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-b67ss_79debca6-ea95-4e3d-8952-5cf5ed2ab970/network-check-target-container/0.log" Sep 09 12:29:03.371169 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:03.371142 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-l6krt_f7ae721c-3b8c-4830-80b9-47e887405da8/iptables-alerter/0.log" Sep 09 12:29:04.028077 ip-10-0-143-139 kubenswrapper[2556]: I0909 12:29:04.028051 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-8pwxh_43684e89-2077-4f0a-8aa8-fe6c149d8c16/tuned/0.log"